Sep 17 01:37:35.055505 ip-10-0-142-18 systemd[1]: Starting Kubernetes Kubelet... Sep 17 01:37:35.435920 ip-10-0-142-18 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:37:35.435920 ip-10-0-142-18 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 01:37:35.435920 ip-10-0-142-18 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:37:35.435920 ip-10-0-142-18 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 01:37:35.435920 ip-10-0-142-18 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:37:35.436434 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.435949 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 01:37:35.440379 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440364 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:35.440379 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440378 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440382 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440385 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440388 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440391 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440393 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440396 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440399 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440401 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440404 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440407 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440409 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440413 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440415 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440418 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440420 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440423 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440426 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440429 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440431 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:35.440441 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440434 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440436 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440439 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440442 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440444 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440447 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440450 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440453 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440458 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440462 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440464 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440467 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440470 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440472 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440475 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440477 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440480 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440483 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440485 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:35.440955 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440488 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440490 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440493 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440495 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440498 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440500 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440503 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440505 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440508 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440510 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440513 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440515 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440518 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440521 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440523 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440526 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440529 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440532 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440534 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440537 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:35.441436 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440539 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440542 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440545 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440547 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440550 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440553 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440556 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440558 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440561 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440566 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440569 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440572 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440574 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440577 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440579 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440582 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440585 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440587 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440590 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440592 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:35.441919 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440595 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440598 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440600 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440604 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440608 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440611 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440616 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440620 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440623 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440626 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440629 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440632 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440634 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440637 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440639 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440642 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440645 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440647 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.440650 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:35.442426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441334 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441341 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441344 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441347 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441350 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441353 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441356 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441358 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441361 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441364 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441366 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441369 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441372 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441374 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441377 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441379 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441383 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441386 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441389 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:35.442895 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441393 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441397 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441401 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441404 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441406 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441410 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441413 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441415 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441418 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441421 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441423 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441426 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441428 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441431 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441434 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441437 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441439 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441443 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441446 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441448 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:35.443421 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441451 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441454 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441456 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441459 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441461 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441464 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441467 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441471 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441474 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441477 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441480 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441482 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441485 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441489 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441492 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441494 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441497 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441500 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441502 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:35.443913 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441505 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441507 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441510 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441514 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441517 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441520 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441524 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441526 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441529 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441532 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441535 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441538 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441541 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441544 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441547 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441550 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441553 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441557 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441560 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441562 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:35.444389 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441565 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441568 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441571 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441573 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441576 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441578 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441581 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441584 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441587 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441589 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441592 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441594 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441596 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441599 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441602 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441604 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441607 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441610 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441612 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441615 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:35.444873 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441617 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441985 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441991 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441994 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.441997 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442000 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442003 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442006 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442008 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442011 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442014 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442017 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442019 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442022 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442024 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442027 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442029 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442032 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442036 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:35.445378 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442039 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442043 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442046 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442049 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442051 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442054 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442057 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442060 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442062 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442065 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442068 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442072 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442075 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442078 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442080 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442083 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442085 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442090 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442093 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:35.445835 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442095 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442098 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442100 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442103 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442106 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442108 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442111 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442113 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442115 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442118 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442120 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442123 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442125 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442128 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442130 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442132 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442135 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442138 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442140 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442143 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:35.446346 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442145 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442148 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442150 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442153 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442155 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442157 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442160 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442162 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442165 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442167 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442169 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442173 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442176 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442179 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442182 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442184 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442187 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442190 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442194 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442197 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:35.446826 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442200 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442203 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442222 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442225 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442227 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442230 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442233 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442235 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442238 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442241 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442243 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442245 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442248 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442251 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442253 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442256 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442258 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442261 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442264 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442266 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:35.447338 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442268 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442271 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442330 2582 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442363 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442370 2582 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442375 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442379 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442382 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442386 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442390 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442393 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442396 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442400 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442403 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442406 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442409 2582 flags.go:64] FLAG: --cgroup-root="" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442412 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442415 2582 flags.go:64] FLAG: --client-ca-file="" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442417 2582 flags.go:64] FLAG: --cloud-provider="external" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442420 2582 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442427 2582 flags.go:64] FLAG: --cluster-domain="" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442430 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442433 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442436 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 01:37:35.447829 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442440 2582 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442443 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442446 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442449 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442452 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442455 2582 flags.go:64] FLAG: --contention-profiling="false" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442458 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442461 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442464 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442467 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442470 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442473 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442476 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442480 2582 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442483 2582 flags.go:64] FLAG: --enable-server="true" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442486 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442490 2582 flags.go:64] FLAG: --event-burst="100" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442494 2582 flags.go:64] FLAG: --event-qps="50" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442497 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442500 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442502 2582 flags.go:64] FLAG: --eviction-hard="" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442506 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442509 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442512 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442515 2582 flags.go:64] FLAG: --eviction-soft="" Sep 17 01:37:35.448419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442518 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442521 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442524 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442528 2582 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442531 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442534 2582 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442537 2582 flags.go:64] FLAG: --feature-gates="" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442541 2582 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442543 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442546 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442549 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442552 2582 flags.go:64] FLAG: --healthz-port="10248" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442555 2582 flags.go:64] FLAG: --help="false" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442558 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442561 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442564 2582 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442567 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442570 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442574 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442578 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442581 2582 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442584 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442587 2582 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 01:37:35.449076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442590 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442593 2582 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442596 2582 flags.go:64] FLAG: --kube-reserved="" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442598 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442601 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442604 2582 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442607 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442610 2582 flags.go:64] FLAG: --lock-file="" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442612 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442615 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442619 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442624 2582 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442628 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442631 2582 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442633 2582 flags.go:64] FLAG: --logging-format="text" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442636 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442639 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442642 2582 flags.go:64] FLAG: --manifest-url="" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442645 2582 flags.go:64] FLAG: --manifest-url-header="" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442649 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442652 2582 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442656 2582 flags.go:64] FLAG: --max-pods="110" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442659 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442662 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442664 2582 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 01:37:35.449776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442667 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442670 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442673 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442677 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442683 2582 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442686 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442690 2582 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442693 2582 flags.go:64] FLAG: --pod-cidr="" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442696 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442701 2582 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442704 2582 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442707 2582 flags.go:64] FLAG: --pods-per-core="0" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442710 2582 flags.go:64] FLAG: --port="10250" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442713 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442716 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0523ed90abc102772" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442719 2582 flags.go:64] FLAG: --qos-reserved="" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442722 2582 flags.go:64] FLAG: --read-only-port="10255" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442725 2582 flags.go:64] FLAG: --register-node="true" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442728 2582 flags.go:64] FLAG: --register-with-taints="" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442732 2582 flags.go:64] FLAG: --registry-burst="10" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442735 2582 flags.go:64] FLAG: --registry-qps="5" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442738 2582 flags.go:64] FLAG: --reserved-cpus="" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442741 2582 flags.go:64] FLAG: --reserved-memory="" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442744 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442747 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 01:37:35.450389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442750 2582 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442753 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442756 2582 flags.go:64] FLAG: --runonce="false" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442759 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442761 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442764 2582 flags.go:64] FLAG: --seccomp-default="false" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442767 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442770 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442773 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442776 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442779 2582 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442783 2582 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442786 2582 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442789 2582 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442792 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442795 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442798 2582 flags.go:64] FLAG: --system-cgroups="" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442801 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442806 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442808 2582 flags.go:64] FLAG: --tls-cert-file="" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442811 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442815 2582 flags.go:64] FLAG: --tls-min-version="" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442818 2582 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442821 2582 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442824 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 01:37:35.451000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442827 2582 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442831 2582 flags.go:64] FLAG: --v="2" Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442838 2582 flags.go:64] FLAG: --version="false" Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442841 2582 flags.go:64] FLAG: --vmodule="" Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442845 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.442848 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442961 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442965 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442968 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442971 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442974 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442977 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442980 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442982 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442985 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442988 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442990 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442993 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442997 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.442999 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443002 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:35.451617 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443006 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443009 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443012 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443015 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443017 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443020 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443023 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443025 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443028 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443031 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443033 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443036 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443040 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443043 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443045 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443048 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443051 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443053 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443056 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443059 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:35.452123 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443061 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443064 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443068 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443072 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443076 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443080 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443082 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443085 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443087 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443091 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443094 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443098 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443101 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443104 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443108 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443111 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443114 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443116 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443119 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:35.452676 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443122 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443124 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443127 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443129 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443132 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443136 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443139 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443141 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443144 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443147 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443149 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443152 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443154 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443157 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443160 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443162 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443165 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443167 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443170 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443172 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:35.453440 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443175 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443177 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443181 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443184 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443187 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443189 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443192 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443195 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443198 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443200 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443203 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443220 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443224 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443227 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443230 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443233 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443236 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443240 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443243 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:35.454062 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443245 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443248 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443251 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443253 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443256 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.443259 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.443956 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.450007 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.450020 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.451280 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.451290 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451344 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451349 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451352 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451355 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:35.454551 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451358 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451360 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451364 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451367 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451369 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451372 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451375 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451377 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451380 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451383 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451386 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451388 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451391 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451393 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451396 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451399 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451401 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451404 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451406 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451410 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:35.454943 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451414 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451417 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451420 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451422 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451425 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451429 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451432 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451435 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451437 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451440 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451442 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451445 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451447 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451450 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451452 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451455 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451458 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451461 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451463 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451466 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:35.455542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451468 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451471 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451473 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451476 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451478 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451481 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451484 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451486 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451489 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451492 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451494 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451497 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451499 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451502 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451505 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451507 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451509 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451513 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451516 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451519 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:35.456031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451521 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451524 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451526 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451530 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451535 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451539 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451542 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451544 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451547 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451550 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451553 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451556 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451558 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451561 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451563 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451566 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451568 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451571 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451573 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:35.456559 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451576 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451578 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451581 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451584 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451586 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451589 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451591 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451594 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451596 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451599 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451602 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451605 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451608 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451611 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451614 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:35.457028 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451616 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.451621 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451723 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451728 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451731 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451733 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451736 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451739 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451742 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451744 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451747 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451750 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451752 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451755 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:37:35.457403 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451758 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451761 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451764 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451767 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451770 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451772 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451775 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451778 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451781 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451784 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451787 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451789 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451792 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451795 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451798 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451801 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451806 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451809 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451811 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451814 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:37:35.457751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451817 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451819 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451822 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451824 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451827 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451830 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451834 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451837 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451840 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451843 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451846 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451849 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451852 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451855 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451857 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451860 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451862 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451865 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451867 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451870 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:37:35.458261 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451873 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451875 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451878 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451881 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451883 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451887 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451890 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451894 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451898 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451901 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451904 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451906 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451909 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451911 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451914 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451917 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451919 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451922 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451924 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:37:35.458755 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451927 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451930 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451932 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451934 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451937 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451939 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451942 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451944 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451947 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451949 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451951 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451954 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451956 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451959 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451961 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451964 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451966 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451969 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451973 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:37:35.459226 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451977 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451980 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451982 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451985 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451989 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451992 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451994 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.451997 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:35.452000 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.452004 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.452090 2582 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.456768 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.457681 2582 server.go:1019] "Starting client certificate rotation" Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.457774 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 01:37:35.459685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.457814 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 01:37:35.460048 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.459832 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 01:37:35.460910 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.460892 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 01:37:35.473533 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.473515 2582 log.go:25] "Validated CRI v1 runtime API" Sep 17 01:37:35.475989 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.475974 2582 log.go:25] "Validated CRI v1 image API" Sep 17 01:37:35.476788 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.476776 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 01:37:35.483013 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.482986 2582 fs.go:135] Filesystem UUIDs: map[0c4e8dee-fccc-4833-8975-349eeb779400:/dev/nvme0n1p3 336fc603-7433-49d8-a4fb-7142aa969a68:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 17 01:37:35.483086 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.483012 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 01:37:35.483666 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.483648 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 01:37:35.488454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.488356 2582 manager.go:216] Machine: {Timestamp:2026-09-17 01:37:35.487274205 +0000 UTC m=+0.322326529 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102230 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2720d6009f1ee130f0fc75b8d4db15 SystemUUID:ec2720d6-009f-1ee1-30f0-fc75b8d4db15 BootID:36b47e5b-ffe6-4f55-adbd-096968a277a6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:fe:4b:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:fe:4b:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:81:a5:7a:cc:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 01:37:35.488454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.488447 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 01:37:35.488574 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.488509 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 01:37:35.491230 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.491193 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 01:37:35.491353 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.491233 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-18.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 01:37:35.491969 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.491960 2582 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 01:37:35.491998 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.491972 2582 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 01:37:35.491998 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.491980 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 01:37:35.493899 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.493889 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 01:37:35.494681 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.494671 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 01:37:35.498325 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498312 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 01:37:35.498435 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498425 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 01:37:35.498467 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498453 2582 kubelet.go:486] "Attempting to sync node with API server" Sep 17 01:37:35.498467 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498465 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 01:37:35.498532 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498485 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 01:37:35.498532 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498497 2582 kubelet.go:398] "Adding apiserver pod source" Sep 17 01:37:35.498532 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498507 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 01:37:35.498815 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498803 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 01:37:35.498873 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.498821 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 01:37:35.501444 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.501430 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 01:37:35.504348 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.504331 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 01:37:35.504417 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.504371 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 01:37:35.506030 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506015 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 01:37:35.506030 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506031 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506038 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506046 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506059 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506065 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506079 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506084 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506091 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506095 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506105 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 01:37:35.506141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506110 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 01:37:35.506818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506808 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 01:37:35.506818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.506817 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 01:37:35.507912 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.507895 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9nd2" Sep 17 01:37:35.509167 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.509143 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-18.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 01:37:35.509281 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.509262 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 01:37:35.509281 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.509268 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 01:37:35.510306 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.510294 2582 server.go:1268] "Started kubelet" Sep 17 01:37:35.510360 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.510327 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 01:37:35.510410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.510371 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 01:37:35.510451 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.510410 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 01:37:35.510961 ip-10-0-142-18 systemd[1]: Started Kubernetes Kubelet. Sep 17 01:37:35.512070 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.512054 2582 server.go:310] "Adding debug handlers to kubelet server" Sep 17 01:37:35.512070 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.512056 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 01:37:35.514720 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.514701 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9nd2" Sep 17 01:37:35.516679 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.516657 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 01:37:35.517747 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.516971 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d5f7f448830b50 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 01:37:35.510268752 +0000 UTC m=+0.345321076,LastTimestamp:2026-09-17 01:37:35.510268752 +0000 UTC m=+0.345321076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}" Sep 17 01:37:35.521076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.521061 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 01:37:35.521153 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.521078 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 01:37:35.521260 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.521246 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:35.522246 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522094 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 01:37:35.522348 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522303 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 01:37:35.522348 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522318 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 01:37:35.522450 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522431 2582 reconstruct.go:97] "Volume reconstruction finished" Sep 17 01:37:35.522450 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522438 2582 reconciler.go:29] "Reconciler: start to sync state" Sep 17 01:37:35.522520 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522482 2582 factory.go:55] Registering systemd factory Sep 17 01:37:35.522520 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522494 2582 factory.go:223] Registration of the systemd container factory successfully Sep 17 01:37:35.522701 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522682 2582 factory.go:153] Registering CRI-O factory Sep 17 01:37:35.522701 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522702 2582 factory.go:223] Registration of the crio container factory successfully Sep 17 01:37:35.522830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522751 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 01:37:35.522830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522768 2582 factory.go:103] Registering Raw factory Sep 17 01:37:35.522830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.522781 2582 manager.go:1196] Started watching for new ooms in manager Sep 17 01:37:35.523096 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.523086 2582 manager.go:349] Starting recovery of all containers Sep 17 01:37:35.524840 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.524821 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:35.527012 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.526991 2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-18.ec2.internal\" not found" node="ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.535706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.535691 2582 manager.go:354] Recovery completed Sep 17 01:37:35.540155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.540143 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.541811 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.541795 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.541880 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.541826 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.541880 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.541839 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.542367 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.542352 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 01:37:35.542439 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.542367 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 01:37:35.542439 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.542368 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.542439 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.542386 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 01:37:35.543117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.543097 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.543195 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.543129 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.543195 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.543141 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.544404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.544389 2582 policy_none.go:49] "None policy: Start" Sep 17 01:37:35.544404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.544405 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 01:37:35.544498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.544417 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 01:37:35.546323 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.546312 2582 policy_none.go:47] "Start" Sep 17 01:37:35.576866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.576853 2582 manager.go:337] "Starting Device Plugin manager" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.576888 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.576899 2582 server.go:87] "Starting device plugin registration server" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.577141 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.577151 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.577443 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.577496 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.577503 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.577962 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 01:37:35.611917 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.578000 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:35.671678 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.671652 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 01:37:35.672779 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.672763 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 01:37:35.672845 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.672807 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 01:37:35.672845 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.672825 2582 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 01:37:35.672937 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.672858 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 01:37:35.674907 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.674889 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:35.677602 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.677588 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.678370 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.678351 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.678462 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.678381 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.678462 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.678394 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.678462 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.678418 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.683487 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.683471 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.683569 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.683494 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-18.ec2.internal\": node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:35.696402 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.696343 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:35.773428 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.773401 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"] Sep 17 01:37:35.773479 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.773466 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.774154 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.774139 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.774238 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.774164 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.774238 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.774174 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.776247 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776236 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.776355 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.776391 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776356 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.776898 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776869 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.776898 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776897 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.777012 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776909 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.777053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.776908 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.777053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.777039 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.777053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.777053 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.779006 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.778992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.779065 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.779014 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:37:35.779592 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.779574 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:37:35.779673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.779603 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:37:35.779673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.779619 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:37:35.793713 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.793695 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-18.ec2.internal\" not found" node="ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.796500 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.796486 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:35.796844 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.796831 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-18.ec2.internal\" not found" node="ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.823419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.823403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6628d7c63703e0aef013a8164e0db91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"f6628d7c63703e0aef013a8164e0db91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.823476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.823426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6628d7c63703e0aef013a8164e0db91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"f6628d7c63703e0aef013a8164e0db91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.823476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.823447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08401cdaa04694297668145e5dc9fe19-config\") pod \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" (UID: \"08401cdaa04694297668145e5dc9fe19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.896709 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.896692 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:35.924033 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.924016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6628d7c63703e0aef013a8164e0db91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"f6628d7c63703e0aef013a8164e0db91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.924091 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.924039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6628d7c63703e0aef013a8164e0db91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"f6628d7c63703e0aef013a8164e0db91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.924091 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.924055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08401cdaa04694297668145e5dc9fe19-config\") pod \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" (UID: \"08401cdaa04694297668145e5dc9fe19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.924172 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.924088 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08401cdaa04694297668145e5dc9fe19-config\") pod \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" (UID: \"08401cdaa04694297668145e5dc9fe19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.924172 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.924102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6628d7c63703e0aef013a8164e0db91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"f6628d7c63703e0aef013a8164e0db91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.924172 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:35.924118 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6628d7c63703e0aef013a8164e0db91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"f6628d7c63703e0aef013a8164e0db91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:35.997283 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:35.997264 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.096936 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.096918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:36.098009 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.097991 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.100064 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.100050 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:36.198951 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.198920 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.301270 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.301200 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.405326 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.405295 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.457395 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.457366 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 01:37:36.457835 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.457493 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 01:37:36.457835 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.457538 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 01:37:36.458144 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.458110 2582 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.142.18:33920->184.193.133.74:6443: use of closed network connection" Sep 17 01:37:36.458209 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.458191 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.142.18:33920->184.193.133.74:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 01:37:36.505875 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.505846 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.521923 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.521889 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 01:32:36 +0000 UTC" deadline="2028-05-03 01:52:42.198747514 +0000 UTC" Sep 17 01:37:36.521923 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.521916 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14256h15m5.676834392s" Sep 17 01:37:36.606957 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.606899 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.709368 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.709335 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.774478 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:36.774439 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6628d7c63703e0aef013a8164e0db91.slice/crio-7fcb7f5500eaadb5657f7fa3125e60fa02f9b07710eceaae2027db014cf57d96 WatchSource:0}: Error finding container 7fcb7f5500eaadb5657f7fa3125e60fa02f9b07710eceaae2027db014cf57d96: Status 404 returned error can't find the container with id 7fcb7f5500eaadb5657f7fa3125e60fa02f9b07710eceaae2027db014cf57d96 Sep 17 01:37:36.775079 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:36.775055 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08401cdaa04694297668145e5dc9fe19.slice/crio-d6a9ce138f7f13598ee00518f3da5667aeb31628530e24b1718d9d39ce596268 WatchSource:0}: Error finding container d6a9ce138f7f13598ee00518f3da5667aeb31628530e24b1718d9d39ce596268: Status 404 returned error can't find the container with id d6a9ce138f7f13598ee00518f3da5667aeb31628530e24b1718d9d39ce596268 Sep 17 01:37:36.777244 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.777204 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:36.779692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:36.779679 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:37:36.810643 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.810624 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:36.917135 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:36.917079 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.018015 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.017993 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.119463 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.119438 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.220146 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.220125 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.320944 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.320918 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.350757 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:37.350734 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:37.421958 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.421928 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.522048 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.521974 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.622917 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.622885 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.678719 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:37.678668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" event={"ID":"08401cdaa04694297668145e5dc9fe19","Type":"ContainerStarted","Data":"d6a9ce138f7f13598ee00518f3da5667aeb31628530e24b1718d9d39ce596268"} Sep 17 01:37:37.679893 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:37.679867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"f6628d7c63703e0aef013a8164e0db91","Type":"ContainerStarted","Data":"7fcb7f5500eaadb5657f7fa3125e60fa02f9b07710eceaae2027db014cf57d96"} Sep 17 01:37:37.711733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:37.711706 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:37.724062 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.724035 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.824775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.824719 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:37.925878 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:37.925849 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.027037 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.027012 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.127917 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.127855 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.228542 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.228510 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.329378 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.329346 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.430070 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.429995 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.530296 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.530265 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.631321 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.631292 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.732354 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:38.732326 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found" Sep 17 01:37:38.796103 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:38.796076 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:37:38.821223 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:38.821185 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:38.832642 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:38.832610 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:37:38.834616 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:38.833782 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:38.840319 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:38.840302 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:37:39.501676 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.501639 2582 apiserver.go:52] "Watching apiserver" Sep 17 01:37:39.510167 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.510142 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 01:37:39.510615 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.510582 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr","openshift-cluster-node-tuning-operator/tuned-fzl8m","openshift-image-registry/node-ca-4687l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal","openshift-network-diagnostics/network-check-target-26n64","openshift-dns/node-resolver-498hn","openshift-multus/multus-66gs2","openshift-multus/multus-additional-cni-plugins-xhrrm","openshift-multus/network-metrics-daemon-vkjcl","openshift-network-operator/iptables-alerter-fg2ms","openshift-ovn-kubernetes/ovnkube-node-xzmdf","kube-system/global-pull-secret-syncer-kjf7c","kube-system/konnectivity-agent-6rdjf","kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"] Sep 17 01:37:39.513453 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.513427 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.515820 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.515798 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.516389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.516362 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.516485 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.516461 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-254nl\"" Sep 17 01:37:39.516538 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.516476 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.516635 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.516616 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 01:37:39.518073 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.518049 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.518318 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.518277 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b9vck\"" Sep 17 01:37:39.518318 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.518305 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 01:37:39.518679 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.518660 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 01:37:39.520454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.520423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.520566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.520552 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.520613 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.520588 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.520649 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.520634 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mc6jn\"" Sep 17 01:37:39.520879 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.520849 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 01:37:39.522473 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.522438 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 01:37:39.522699 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.522677 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mdvvn\"" Sep 17 01:37:39.522875 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.522841 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:39.522972 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.522924 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:39.523039 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.522981 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 01:37:39.523150 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.523128 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 01:37:39.523222 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.523148 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.523222 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.523160 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.523222 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.523179 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 01:37:39.525406 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.525383 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.527485 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.527463 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.527598 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.527574 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jsb6n\"" Sep 17 01:37:39.527660 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.527576 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.527768 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.527732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:39.527942 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.527916 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:39.530525 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.530500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.533480 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.532719 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.534153 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.534113 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2bsqq\"" Sep 17 01:37:39.534959 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.534532 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 01:37:39.535075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.535043 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 01:37:39.535693 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.535656 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlcpq\"" Sep 17 01:37:39.535693 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.535685 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.535889 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.535757 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.536783 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.536749 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.538768 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.538747 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 01:37:39.539024 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.539007 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.539101 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.539078 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.539189 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.539166 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zwql2\"" Sep 17 01:37:39.539493 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.539471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.541687 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.541649 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 01:37:39.541796 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.541758 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 01:37:39.541894 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.541871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.541986 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.541962 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:39.542278 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.542254 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 01:37:39.542278 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.542271 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 01:37:39.542400 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.542326 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 01:37:39.542400 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.542392 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 01:37:39.542560 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.542542 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w6c5b\"" Sep 17 01:37:39.548674 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zfx\" (UniqueName: \"kubernetes.io/projected/bde4c47b-878a-400d-a28b-4eb30e925a9f-kube-api-access-f9zfx\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.548779 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-cnibin\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.548779 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.548779 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.548779 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f1e63a-3dcd-4a33-8ab4-0213f5156727-host-slash\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqcx\" (UniqueName: \"kubernetes.io/projected/46f1e63a-3dcd-4a33-8ab4-0213f5156727-kube-api-access-mdqcx\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/502a29ca-ecd3-481a-b86d-f66314d15ae1-konnectivity-ca\") pod \"konnectivity-agent-6rdjf\" (UID: \"502a29ca-ecd3-481a-b86d-f66314d15ae1\") " pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c84a533-aeac-4d80-b820-f72b570c337d-host\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6v5k\" (UniqueName: \"kubernetes.io/projected/7c84a533-aeac-4d80-b820-f72b570c337d-kube-api-access-v6v5k\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c84a533-aeac-4d80-b820-f72b570c337d-serviceca\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-system-cni-dir\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.548987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.549016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.549012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f1e63a-3dcd-4a33-8ab4-0213f5156727-iptables-alerter-script\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.549459 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.549038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/502a29ca-ecd3-481a-b86d-f66314d15ae1-agent-certs\") pod \"konnectivity-agent-6rdjf\" (UID: \"502a29ca-ecd3-481a-b86d-f66314d15ae1\") " pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.553387 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.553368 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-78dqh" Sep 17 01:37:39.562674 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.562653 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-78dqh" Sep 17 01:37:39.623773 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.623753 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 01:37:39.649941 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.649922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-run\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.649944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-tuned\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.649963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-tmp-dir\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.649979 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysctl-d\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.649992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-systemd\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-sys\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-lib-modules\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.650061 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67nqw\" (UniqueName: \"kubernetes.io/projected/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-kube-api-access-67nqw\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-cni-netd\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovnkube-config\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/502a29ca-ecd3-481a-b86d-f66314d15ae1-agent-certs\") pod \"konnectivity-agent-6rdjf\" (UID: \"502a29ca-ecd3-481a-b86d-f66314d15ae1\") " pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cb5\" (UniqueName: \"kubernetes.io/projected/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-kube-api-access-c9cb5\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-kubelet\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-etc-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650199 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-device-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-cnibin\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650306 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-systemd\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.650320 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-log-socket\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysconfig\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-socket-dir-parent\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-ovn\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-system-cni-dir\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650502 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650505 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 01:37:39.651056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-system-cni-dir\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.650554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-k8s-cni-cncf-io\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651168 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-netns\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-env-overrides\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651219 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbb95db5-3662-41bf-873e-e2e2e34947fc-kubelet-config\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zfx\" (UniqueName: \"kubernetes.io/projected/bde4c47b-878a-400d-a28b-4eb30e925a9f-kube-api-access-f9zfx\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbckz\" (UniqueName: \"kubernetes.io/projected/9b9ad765-86b9-4f5c-89e6-9decb263dffb-kube-api-access-qbckz\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-conf-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651303 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-kubelet\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c84a533-aeac-4d80-b820-f72b570c337d-host\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c84a533-aeac-4d80-b820-f72b570c337d-host\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6v5k\" (UniqueName: \"kubernetes.io/projected/7c84a533-aeac-4d80-b820-f72b570c337d-kube-api-access-v6v5k\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.651457 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651507 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-cni-bin\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbb95db5-3662-41bf-873e-e2e2e34947fc-dbus\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f1e63a-3dcd-4a33-8ab4-0213f5156727-iptables-alerter-script\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-hosts-file\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887s8\" (UniqueName: \"kubernetes.io/projected/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-kube-api-access-887s8\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651655 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovnkube-script-lib\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-sys-fs\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651704 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26831bc7-27ea-414d-9685-58b9e1d017d8-cni-binary-copy\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651729 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9ad765-86b9-4f5c-89e6-9decb263dffb-tmp\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651866 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-cni-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysctl-conf\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.652057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.651959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-cnibin\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f1e63a-3dcd-4a33-8ab4-0213f5156727-host-slash\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bde4c47b-878a-400d-a28b-4eb30e925a9f-cnibin\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqcx\" (UniqueName: \"kubernetes.io/projected/46f1e63a-3dcd-4a33-8ab4-0213f5156727-kube-api-access-mdqcx\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46f1e63a-3dcd-4a33-8ab4-0213f5156727-iptables-alerter-script\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/502a29ca-ecd3-481a-b86d-f66314d15ae1-konnectivity-ca\") pod \"konnectivity-agent-6rdjf\" (UID: \"502a29ca-ecd3-481a-b86d-f66314d15ae1\") " pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f1e63a-3dcd-4a33-8ab4-0213f5156727-host-slash\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-kubernetes\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-socket-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-var-lib-kubelet\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-host\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-etc-kubernetes\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glwmz\" (UniqueName: \"kubernetes.io/projected/26831bc7-27ea-414d-9685-58b9e1d017d8-kube-api-access-glwmz\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-slash\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-var-lib-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c84a533-aeac-4d80-b820-f72b570c337d-serviceca\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.652819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-daemon-config\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-systemd-units\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/502a29ca-ecd3-481a-b86d-f66314d15ae1-konnectivity-ca\") pod \"konnectivity-agent-6rdjf\" (UID: \"502a29ca-ecd3-481a-b86d-f66314d15ae1\") " pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-cni-bin\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovn-node-metrics-cert\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-cni-multus\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-hostroot\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652820 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-multus-certs\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-node-log\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652881 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psb6r\" (UniqueName: \"kubernetes.io/projected/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-kube-api-access-psb6r\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.652967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-modprobe-d\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.653061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-registration-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.653084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-system-cni-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.653099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-run-netns\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.653624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.653115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.654410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.653700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.654410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.653871 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c84a533-aeac-4d80-b820-f72b570c337d-serviceca\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.654410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.654106 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bde4c47b-878a-400d-a28b-4eb30e925a9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.656767 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.656743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/502a29ca-ecd3-481a-b86d-f66314d15ae1-agent-certs\") pod \"konnectivity-agent-6rdjf\" (UID: \"502a29ca-ecd3-481a-b86d-f66314d15ae1\") " pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.660270 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.660249 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zfx\" (UniqueName: \"kubernetes.io/projected/bde4c47b-878a-400d-a28b-4eb30e925a9f-kube-api-access-f9zfx\") pod \"multus-additional-cni-plugins-xhrrm\" (UID: \"bde4c47b-878a-400d-a28b-4eb30e925a9f\") " pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.660370 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.660350 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqcx\" (UniqueName: \"kubernetes.io/projected/46f1e63a-3dcd-4a33-8ab4-0213f5156727-kube-api-access-mdqcx\") pod \"iptables-alerter-fg2ms\" (UID: \"46f1e63a-3dcd-4a33-8ab4-0213f5156727\") " pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.662350 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.662327 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:39.662420 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.662353 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:39.662420 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.662367 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:39.662497 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.662428 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:40.162402392 +0000 UTC m=+4.997454723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:39.664001 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.663981 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6v5k\" (UniqueName: \"kubernetes.io/projected/7c84a533-aeac-4d80-b820-f72b570c337d-kube-api-access-v6v5k\") pod \"node-ca-4687l\" (UID: \"7c84a533-aeac-4d80-b820-f72b570c337d\") " pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.684982 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.684961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" event={"ID":"08401cdaa04694297668145e5dc9fe19","Type":"ContainerStarted","Data":"4bed9a7bf3c8a72247a40be1979536ded04b38485ee6d65a9f12e06acc93a2fd"} Sep 17 01:37:39.685123 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.685092 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:39.686427 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.686410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"f6628d7c63703e0aef013a8164e0db91","Type":"ContainerStarted","Data":"fcee55c9c499cb3d4f689cd806851c108933e5a84b16104ada93704175a331d7"} Sep 17 01:37:39.686581 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.686567 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:39.697615 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.697599 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:37:39.697666 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.697641 2582 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" Sep 17 01:37:39.697939 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.697922 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:37:39.698020 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.698006 2582 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" Sep 17 01:37:39.754041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.753993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-socket-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.754041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-var-lib-kubelet\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.754041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-host\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754046 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-etc-kubernetes\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glwmz\" (UniqueName: \"kubernetes.io/projected/26831bc7-27ea-414d-9685-58b9e1d017d8-kube-api-access-glwmz\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-slash\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-var-lib-kubelet\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-var-lib-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-etc-kubernetes\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-socket-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-var-lib-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-daemon-config\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754171 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-host\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-systemd-units\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754273 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-slash\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-cni-bin\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-systemd-units\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-cni-bin\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovn-node-metrics-cert\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-cni-multus\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-hostroot\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754415 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-multus-certs\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-cni-multus\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-node-log\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-hostroot\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psb6r\" (UniqueName: \"kubernetes.io/projected/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-kube-api-access-psb6r\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.754578 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-modprobe-d\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-node-log\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-registration-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-multus-certs\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-system-cni-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754596 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-registration-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-run-netns\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754643 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-daemon-config\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-run\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754658 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-modprobe-d\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-tuned\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754691 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-system-cni-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-tmp-dir\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-run\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-run-netns\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysctl-d\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-systemd\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.755460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-sys\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-lib-modules\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysctl-d\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67nqw\" (UniqueName: \"kubernetes.io/projected/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-kube-api-access-67nqw\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754871 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-systemd\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-cni-netd\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovnkube-config\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754889 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-sys\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cb5\" (UniqueName: \"kubernetes.io/projected/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-kube-api-access-c9cb5\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754954 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-tmp-dir\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-kubelet\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-lib-modules\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.754988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-etc-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-cni-netd\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-etc-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-kubelet\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-device-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.756276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-cnibin\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-systemd\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.755253 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-device-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-openvswitch\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.755327 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:37:40.255291006 +0000 UTC m=+5.090343319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755339 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-cnibin\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755350 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-log-socket\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-systemd\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysconfig\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-log-socket\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-socket-dir-parent\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-ovn\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysconfig\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-k8s-cni-cncf-io\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755522 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-run-ovn\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.756954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-k8s-cni-cncf-io\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-netns\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-socket-dir-parent\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755563 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-run-netns\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-env-overrides\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbb95db5-3662-41bf-873e-e2e2e34947fc-kubelet-config\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbckz\" (UniqueName: \"kubernetes.io/projected/9b9ad765-86b9-4f5c-89e6-9decb263dffb-kube-api-access-qbckz\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755648 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovnkube-config\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-conf-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbb95db5-3662-41bf-873e-e2e2e34947fc-kubelet-config\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-kubelet\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-conf-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-cni-bin\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbb95db5-3662-41bf-873e-e2e2e34947fc-dbus\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-hosts-file\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.755860 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:39.757486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887s8\" (UniqueName: \"kubernetes.io/projected/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-kube-api-access-887s8\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovnkube-script-lib\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-kubelet\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755867 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-host-var-lib-cni-bin\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-sys-fs\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-env-overrides\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbb95db5-3662-41bf-873e-e2e2e34947fc-dbus\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.755961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26831bc7-27ea-414d-9685-58b9e1d017d8-cni-binary-copy\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:39.755993 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:40.255972597 +0000 UTC m=+5.091024910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-sys-fs\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9ad765-86b9-4f5c-89e6-9decb263dffb-tmp\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-hosts-file\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.758004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756277 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-cni-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysctl-conf\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-kubernetes\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26831bc7-27ea-414d-9685-58b9e1d017d8-multus-cni-dir\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-kubernetes\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovnkube-script-lib\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26831bc7-27ea-414d-9685-58b9e1d017d8-cni-binary-copy\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756570 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-sysctl-conf\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.756761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-ovn-node-metrics-cert\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.757657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b9ad765-86b9-4f5c-89e6-9decb263dffb-etc-tuned\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.758498 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.758094 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9ad765-86b9-4f5c-89e6-9decb263dffb-tmp\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.764626 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.764603 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67nqw\" (UniqueName: \"kubernetes.io/projected/ae2b81bf-c42e-4fbb-b8f4-513afd095ee5-kube-api-access-67nqw\") pod \"aws-ebs-csi-driver-node-j2vsr\" (UID: \"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.765819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.765784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbckz\" (UniqueName: \"kubernetes.io/projected/9b9ad765-86b9-4f5c-89e6-9decb263dffb-kube-api-access-qbckz\") pod \"tuned-fzl8m\" (UID: \"9b9ad765-86b9-4f5c-89e6-9decb263dffb\") " pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.765920 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.765833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psb6r\" (UniqueName: \"kubernetes.io/projected/2369bafd-2c89-4d72-ac21-0c5ba8f727e7-kube-api-access-psb6r\") pod \"ovnkube-node-xzmdf\" (UID: \"2369bafd-2c89-4d72-ac21-0c5ba8f727e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.766255 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.766210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887s8\" (UniqueName: \"kubernetes.io/projected/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-kube-api-access-887s8\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:39.767606 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.767585 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glwmz\" (UniqueName: \"kubernetes.io/projected/26831bc7-27ea-414d-9685-58b9e1d017d8-kube-api-access-glwmz\") pod \"multus-66gs2\" (UID: \"26831bc7-27ea-414d-9685-58b9e1d017d8\") " pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.767678 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.767657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cb5\" (UniqueName: \"kubernetes.io/projected/5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0-kube-api-access-c9cb5\") pod \"node-resolver-498hn\" (UID: \"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0\") " pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.825906 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.825883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fg2ms" Sep 17 01:37:39.833510 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.833421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:37:39.836002 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.835982 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f1e63a_3dcd_4a33_8ab4_0213f5156727.slice/crio-007234bed365570bd4f863558c39ebf85ea2f7bbb9b45abde260e3266c0fcada WatchSource:0}: Error finding container 007234bed365570bd4f863558c39ebf85ea2f7bbb9b45abde260e3266c0fcada: Status 404 returned error can't find the container with id 007234bed365570bd4f863558c39ebf85ea2f7bbb9b45abde260e3266c0fcada Sep 17 01:37:39.842844 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.842824 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502a29ca_ecd3_481a_b86d_f66314d15ae1.slice/crio-aada90709a79988d961b299f811e9bdcd61b1382d1ea24707550b40ccc00c853 WatchSource:0}: Error finding container aada90709a79988d961b299f811e9bdcd61b1382d1ea24707550b40ccc00c853: Status 404 returned error can't find the container with id aada90709a79988d961b299f811e9bdcd61b1382d1ea24707550b40ccc00c853 Sep 17 01:37:39.843387 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.843365 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4687l" Sep 17 01:37:39.849887 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.849865 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" Sep 17 01:37:39.852480 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.852461 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c84a533_aeac_4d80_b820_f72b570c337d.slice/crio-329a25cc0769c008320b5fed9dfaf114239d5b529e30851372118c03514ab7ac WatchSource:0}: Error finding container 329a25cc0769c008320b5fed9dfaf114239d5b529e30851372118c03514ab7ac: Status 404 returned error can't find the container with id 329a25cc0769c008320b5fed9dfaf114239d5b529e30851372118c03514ab7ac Sep 17 01:37:39.856604 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.856584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-498hn" Sep 17 01:37:39.859782 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.859762 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde4c47b_878a_400d_a28b_4eb30e925a9f.slice/crio-b40f9c7a30ce80447374be41ba62a93666691d4754d4d7699ee1e7b8afa26273 WatchSource:0}: Error finding container b40f9c7a30ce80447374be41ba62a93666691d4754d4d7699ee1e7b8afa26273: Status 404 returned error can't find the container with id b40f9c7a30ce80447374be41ba62a93666691d4754d4d7699ee1e7b8afa26273 Sep 17 01:37:39.862468 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.862452 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-66gs2" Sep 17 01:37:39.866031 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.866006 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c521d6a_4e1e_4e0f_8e8f_21289c64a2a0.slice/crio-4c6fc6c0ee38e6f616362e1608289e251a003c7bfe0d68d7abe227c60fb9208d WatchSource:0}: Error finding container 4c6fc6c0ee38e6f616362e1608289e251a003c7bfe0d68d7abe227c60fb9208d: Status 404 returned error can't find the container with id 4c6fc6c0ee38e6f616362e1608289e251a003c7bfe0d68d7abe227c60fb9208d Sep 17 01:37:39.867798 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.867785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" Sep 17 01:37:39.874323 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.874300 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" Sep 17 01:37:39.877531 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.877508 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26831bc7_27ea_414d_9685_58b9e1d017d8.slice/crio-3d72fe1c632a2f7abc21e6dc7dfeabe059d43bca6a963f03d1f3b216f4cbb92d WatchSource:0}: Error finding container 3d72fe1c632a2f7abc21e6dc7dfeabe059d43bca6a963f03d1f3b216f4cbb92d: Status 404 returned error can't find the container with id 3d72fe1c632a2f7abc21e6dc7dfeabe059d43bca6a963f03d1f3b216f4cbb92d Sep 17 01:37:39.878710 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:39.878695 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:37:39.886128 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.886107 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9ad765_86b9_4f5c_89e6_9decb263dffb.slice/crio-81e1700190b1bdd43ce4b8912c386b08b703359a3e9acfeb6e0ecd4fc1daf8be WatchSource:0}: Error finding container 81e1700190b1bdd43ce4b8912c386b08b703359a3e9acfeb6e0ecd4fc1daf8be: Status 404 returned error can't find the container with id 81e1700190b1bdd43ce4b8912c386b08b703359a3e9acfeb6e0ecd4fc1daf8be Sep 17 01:37:39.890265 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.890227 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2b81bf_c42e_4fbb_b8f4_513afd095ee5.slice/crio-ddc721f3b456154d94d2bb948e4729e370c81e417fb9a29f898536375f7eeece WatchSource:0}: Error finding container ddc721f3b456154d94d2bb948e4729e370c81e417fb9a29f898536375f7eeece: Status 404 returned error can't find the container with id ddc721f3b456154d94d2bb948e4729e370c81e417fb9a29f898536375f7eeece Sep 17 01:37:39.894686 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:37:39.894666 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2369bafd_2c89_4d72_ac21_0c5ba8f727e7.slice/crio-6a2408165ea015e5a5047f6442b8c0b01e314ddd3be130868b36ed485d191313 WatchSource:0}: Error finding container 6a2408165ea015e5a5047f6442b8c0b01e314ddd3be130868b36ed485d191313: Status 404 returned error can't find the container with id 6a2408165ea015e5a5047f6442b8c0b01e314ddd3be130868b36ed485d191313 Sep 17 01:37:40.260531 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.260491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:40.260644 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.260547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:40.260644 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.260590 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:40.260775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260651 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:40.260775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260707 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:40.260775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260719 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:40.260775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260736 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:40.260775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260749 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:40.260775 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260717 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:37:41.260696901 +0000 UTC m=+6.095749232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:40.260996 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260783 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:41.260765012 +0000 UTC m=+6.095817332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:40.260996 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.260805 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:41.260793227 +0000 UTC m=+6.095845541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:40.566577 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.566507 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:32:39 +0000 UTC" deadline="2028-02-24 07:14:54.94169243 +0000 UTC" Sep 17 01:37:40.566577 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.566578 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12605h37m14.375119034s" Sep 17 01:37:40.673596 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.673092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:40.673596 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.673211 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:40.673596 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.673091 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:40.673596 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:40.673382 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:40.728658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.728621 2582 generic.go:356] "Generic (PLEG): container finished" podID="f6628d7c63703e0aef013a8164e0db91" containerID="fcee55c9c499cb3d4f689cd806851c108933e5a84b16104ada93704175a331d7" exitCode=0 Sep 17 01:37:40.728819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.728716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"f6628d7c63703e0aef013a8164e0db91","Type":"ContainerDied","Data":"fcee55c9c499cb3d4f689cd806851c108933e5a84b16104ada93704175a331d7"} Sep 17 01:37:40.741692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.741087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"6a2408165ea015e5a5047f6442b8c0b01e314ddd3be130868b36ed485d191313"} Sep 17 01:37:40.745521 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.745463 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" podStartSLOduration=1.745444361 podStartE2EDuration="1.745444361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:39.759106589 +0000 UTC m=+4.594158928" watchObservedRunningTime="2026-09-17 01:37:40.745444361 +0000 UTC m=+5.580496700" Sep 17 01:37:40.753655 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.753626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" event={"ID":"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5","Type":"ContainerStarted","Data":"ddc721f3b456154d94d2bb948e4729e370c81e417fb9a29f898536375f7eeece"} Sep 17 01:37:40.769332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.769304 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-66gs2" event={"ID":"26831bc7-27ea-414d-9685-58b9e1d017d8","Type":"ContainerStarted","Data":"3d72fe1c632a2f7abc21e6dc7dfeabe059d43bca6a963f03d1f3b216f4cbb92d"} Sep 17 01:37:40.776091 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.771745 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-498hn" event={"ID":"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0","Type":"ContainerStarted","Data":"4c6fc6c0ee38e6f616362e1608289e251a003c7bfe0d68d7abe227c60fb9208d"} Sep 17 01:37:40.776091 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.775951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerStarted","Data":"b40f9c7a30ce80447374be41ba62a93666691d4754d4d7699ee1e7b8afa26273"} Sep 17 01:37:40.780832 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.780803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4687l" event={"ID":"7c84a533-aeac-4d80-b820-f72b570c337d","Type":"ContainerStarted","Data":"329a25cc0769c008320b5fed9dfaf114239d5b529e30851372118c03514ab7ac"} Sep 17 01:37:40.801001 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.800973 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" event={"ID":"9b9ad765-86b9-4f5c-89e6-9decb263dffb","Type":"ContainerStarted","Data":"81e1700190b1bdd43ce4b8912c386b08b703359a3e9acfeb6e0ecd4fc1daf8be"} Sep 17 01:37:40.806547 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.806519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6rdjf" event={"ID":"502a29ca-ecd3-481a-b86d-f66314d15ae1","Type":"ContainerStarted","Data":"aada90709a79988d961b299f811e9bdcd61b1382d1ea24707550b40ccc00c853"} Sep 17 01:37:40.834636 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:40.834601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fg2ms" event={"ID":"46f1e63a-3dcd-4a33-8ab4-0213f5156727","Type":"ContainerStarted","Data":"007234bed365570bd4f863558c39ebf85ea2f7bbb9b45abde260e3266c0fcada"} Sep 17 01:37:41.268889 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.268496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:41.269053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.268891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:41.269053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.268945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:41.269181 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269088 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:41.269181 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269155 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:37:43.269135282 +0000 UTC m=+8.104187613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:41.269600 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269580 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:41.269675 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269640 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:43.269623789 +0000 UTC m=+8.104676121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:41.269746 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269721 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:41.269803 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269759 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:41.269803 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269772 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:41.269907 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.269813 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:43.269799754 +0000 UTC m=+8.104852082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:41.567504 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.567381 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:32:39 +0000 UTC" deadline="2028-06-03 16:03:42.551282944 +0000 UTC" Sep 17 01:37:41.567504 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.567421 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15014h26m0.983866648s" Sep 17 01:37:41.675565 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.675525 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:41.675763 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:41.675662 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:41.846618 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:41.846497 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"f6628d7c63703e0aef013a8164e0db91","Type":"ContainerStarted","Data":"0af74dcd7ddaf3195ccba324c1a2fdca66fbc21c83b9638b0355c9fd2a8ec16b"} Sep 17 01:37:42.673613 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:42.673583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:42.674200 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:42.673724 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:42.674200 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:42.674090 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:42.674200 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:42.674181 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:43.285739 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:43.285699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:43.285931 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:43.285755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:43.285931 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:43.285787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:43.285931 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.285889 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.285950 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.285970 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:37:47.285948805 +0000 UTC m=+12.121001140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.285972 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.285997 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.286033 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.286044 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:47.286032298 +0000 UTC m=+12.121084616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:43.286107 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.286079 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:47.28606407 +0000 UTC m=+12.121116382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:43.674050 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:43.673966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:43.674565 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:43.674111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:44.673412 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:44.673322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:44.673574 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:44.673551 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:44.673684 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:44.673661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:44.673812 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:44.673789 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:45.674173 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:45.674143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:45.674687 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:45.674265 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:46.673505 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:46.673468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:46.673682 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:46.673602 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:46.673682 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:46.673657 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:46.673804 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:46.673776 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:47.321514 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:47.321249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:47.321514 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:47.321318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:47.321514 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:47.321360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:47.321514 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321437 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:47.321514 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321501 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:47.321514 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321440 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:47.322240 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321545 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:47.322240 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321556 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:47.322240 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321665 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:55.321478824 +0000 UTC m=+20.156531151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:47.322240 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321693 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:37:55.321677723 +0000 UTC m=+20.156730045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:47.322240 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.321711 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:37:55.321700461 +0000 UTC m=+20.156752787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:47.673505 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:47.673413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:47.673658 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:47.673544 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:48.673475 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:48.673397 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:48.673935 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:48.673535 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:48.673935 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:48.673595 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:48.673935 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:48.673699 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:49.673957 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:49.673924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:49.674367 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:49.674056 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:50.673593 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:50.673566 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:50.673754 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:50.673667 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:50.673754 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:50.673715 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:50.673855 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:50.673814 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:51.673830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:51.673789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:51.674292 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:51.673908 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:52.673097 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:52.673060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:52.673379 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:52.673056 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:52.673379 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:52.673204 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:52.673379 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:52.673253 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:53.673473 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:53.673431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:53.673993 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:53.673580 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:54.673146 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:54.673107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:54.673362 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:54.673107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:54.673362 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:54.673267 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:54.673362 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:54.673290 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:55.377836 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:55.377799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:55.377850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:55.377889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.377972 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.378003 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.378016 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.377976 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.377977 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.378074 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:11.378057579 +0000 UTC m=+36.213109894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.378159 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:11.37814489 +0000 UTC m=+36.213197203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:37:55.378319 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.378180 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:38:11.378171352 +0000 UTC m=+36.213223668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:37:55.674466 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:55.674385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:55.674643 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:55.674512 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:56.673187 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:56.673155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:56.673187 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:56.673195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:56.673666 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:56.673298 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:56.673666 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:56.673445 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:57.673203 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:57.673166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:57.673643 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:57.673308 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:58.673542 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.673327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:37:58.674176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.673375 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:37:58.674176 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:58.673664 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:37:58.674176 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:58.673676 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:37:58.884292 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.884250 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6rdjf" event={"ID":"502a29ca-ecd3-481a-b86d-f66314d15ae1","Type":"ContainerStarted","Data":"271fbef5763b9a04bf11e94f5ec13ef9ba0dcba9576a747be6c7226ebc14d746"} Sep 17 01:37:58.887482 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.887455 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"ef01116f0da72d6127afcebea2a2608d0904e24ed3dbcd4200d39b046a777e8b"} Sep 17 01:37:58.887622 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.887535 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"18ecbab9b4422e30c67a403dfa3e4ac300575db2172b94d71fa60a827f8ec503"} Sep 17 01:37:58.887622 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.887585 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"8789898f9589240dff4e503514d1b7d1adc4c876bd0292d2505859092eed2d84"} Sep 17 01:37:58.887715 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.887633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"38dcd3b96696ba1743698d5036ecee1ba61aac4b4e7c68bc077ff4d79ed16de1"} Sep 17 01:37:58.887715 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.887680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"fbe9ecd3703fd6f63d163fa9e90f3e77fab4f712f5a60e53a45ff619ec37936f"} Sep 17 01:37:58.887806 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.887727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"0c0a19698710933d840baa03947282e870bc4383fb0cca4316735d0cc2ce4490"} Sep 17 01:37:58.889193 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.889171 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" event={"ID":"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5","Type":"ContainerStarted","Data":"5d4683a641195257853533dd36762a9b7424ea018459b1e2516111047e384710"} Sep 17 01:37:58.890741 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.890722 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-66gs2" event={"ID":"26831bc7-27ea-414d-9685-58b9e1d017d8","Type":"ContainerStarted","Data":"58ecd72569cda73fcec1d099417507a3013dd3ad5f0d958acfd8ab42c7d66b74"} Sep 17 01:37:58.892250 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.892194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-498hn" event={"ID":"5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0","Type":"ContainerStarted","Data":"1508c5ccbd29b1551c7bed728c86896197691901e09d452b691eef56c72e7591"} Sep 17 01:37:58.893770 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.893749 2582 generic.go:356] "Generic (PLEG): container finished" podID="bde4c47b-878a-400d-a28b-4eb30e925a9f" containerID="d5ca01f1c9807754396a1e23bdd27d946810bf26f3245fac844a7ac87f542b4a" exitCode=0 Sep 17 01:37:58.893871 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.893809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerDied","Data":"d5ca01f1c9807754396a1e23bdd27d946810bf26f3245fac844a7ac87f542b4a"} Sep 17 01:37:58.895447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.895372 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4687l" event={"ID":"7c84a533-aeac-4d80-b820-f72b570c337d","Type":"ContainerStarted","Data":"2846d4f6e135362bd75c1e1f6acb41f21d1be99d44f429683a3a6b3cced9ab38"} Sep 17 01:37:58.896966 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.896946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" event={"ID":"9b9ad765-86b9-4f5c-89e6-9decb263dffb","Type":"ContainerStarted","Data":"328ec32bc7b969931c13ac25999e852ae96b1712372aea3e91e0a275dc685813"} Sep 17 01:37:58.897440 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.897406 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" podStartSLOduration=19.897395122 podStartE2EDuration="19.897395122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:41.86360306 +0000 UTC m=+6.698655397" watchObservedRunningTime="2026-09-17 01:37:58.897395122 +0000 UTC m=+23.732447456" Sep 17 01:37:58.897899 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.897876 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6rdjf" podStartSLOduration=12.104642335 podStartE2EDuration="22.897868083s" totalImagesPullingTime="10.793225748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:58.897589534 +0000 UTC m=+23.732641866" watchObservedRunningTime="2026-09-17 01:37:58.897868083 +0000 UTC m=+23.732920416" Sep 17 01:37:58.924236 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.924179 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:37:58.944940 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.944899 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-498hn" podStartSLOduration=4.9799969950000005 podStartE2EDuration="22.944882881s" totalImagesPullingTime="17.964885886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:58.928037474 +0000 UTC m=+23.763089808" watchObservedRunningTime="2026-09-17 01:37:58.944882881 +0000 UTC m=+23.779935215" Sep 17 01:37:58.945053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.944987 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-66gs2" podStartSLOduration=4.943108596 podStartE2EDuration="22.9449806s" totalImagesPullingTime="18.001872004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:58.944257626 +0000 UTC m=+23.779309960" watchObservedRunningTime="2026-09-17 01:37:58.9449806 +0000 UTC m=+23.780032935" Sep 17 01:37:58.959303 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:58.959266 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fzl8m" podStartSLOduration=5.014347063 podStartE2EDuration="22.959253914s" totalImagesPullingTime="17.944906851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:58.959035062 +0000 UTC m=+23.794087392" watchObservedRunningTime="2026-09-17 01:37:58.959253914 +0000 UTC m=+23.794306247" Sep 17 01:37:59.005955 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.005932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:37:59.101916 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.101835 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:37:59.202347 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.202312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:37:59.469424 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.469403 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 01:37:59.591013 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.590911 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T01:37:59.469419471Z","UUID":"e2682518-d0d2-4792-8f37-0bada58406b4","Handler":null,"Name":"","Endpoint":""} Sep 17 01:37:59.592229 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.592195 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 01:37:59.592344 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.592242 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 01:37:59.673977 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.673918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:37:59.674404 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:37:59.674016 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:37:59.900478 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.900438 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fg2ms" event={"ID":"46f1e63a-3dcd-4a33-8ab4-0213f5156727","Type":"ContainerStarted","Data":"53f92877066aa7208b292c7f63deb94d1d178c4f7c3eea3c0a7d11a1fc0138da"} Sep 17 01:37:59.902265 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.902234 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" event={"ID":"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5","Type":"ContainerStarted","Data":"d8b5c03b79769d2936fb96b4fe658fcd00e153c22b0d2495d0413ab11275850a"} Sep 17 01:37:59.913768 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.913717 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4687l" podStartSLOduration=5.935000579 podStartE2EDuration="23.913702828s" totalImagesPullingTime="17.978702249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:58.971263361 +0000 UTC m=+23.806315694" watchObservedRunningTime="2026-09-17 01:37:59.913702828 +0000 UTC m=+24.748755212" Sep 17 01:37:59.914112 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:37:59.914081 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fg2ms" podStartSLOduration=5.921165991 podStartE2EDuration="23.914074177s" totalImagesPullingTime="17.992908186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:59.91362113 +0000 UTC m=+24.748673462" watchObservedRunningTime="2026-09-17 01:37:59.914074177 +0000 UTC m=+24.749126509" Sep 17 01:38:00.030157 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.030130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:00.131673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.131651 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:00.673977 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.673951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:00.674375 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.673989 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:00.674375 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:00.674062 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:00.674375 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:00.674114 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:00.807820 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.807787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:00.908104 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.908072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:00.909194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:00.909170 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"e771ea2f93dfb94d38eb339ddefc3b803131084406790017472b4574124b377d"} Sep 17 01:38:01.663026 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:01.662994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:01.673872 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:01.673840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:01.673997 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:01.673958 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:38:01.913947 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:01.913877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" event={"ID":"ae2b81bf-c42e-4fbb-b8f4-513afd095ee5","Type":"ContainerStarted","Data":"295f206448d6430c938987ca9e27f7dc0b504f1eda87e9ba585343d459ff6a72"} Sep 17 01:38:01.931454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:01.931406 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2vsr" podStartSLOduration=5.117257192 podStartE2EDuration="25.931387241s" totalImagesPullingTime="20.814130049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:01.930929543 +0000 UTC m=+26.765981876" watchObservedRunningTime="2026-09-17 01:38:01.931387241 +0000 UTC m=+26.766439575" Sep 17 01:38:02.673074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.673037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:02.673265 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.673037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:02.673265 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:02.673157 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:02.673387 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:02.673272 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:02.909960 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.909930 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:38:02.910712 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.910686 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:38:02.915936 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.915908 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:38:02.916235 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.916198 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6rdjf" Sep 17 01:38:02.980191 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:02.980151 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:03.673502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.673342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:03.673639 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:03.673585 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:38:03.920888 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.920829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" event={"ID":"2369bafd-2c89-4d72-ac21-0c5ba8f727e7","Type":"ContainerStarted","Data":"84bd687da7c219edca6136ace226cb935b0f8d556e0f306823cee1d1e17a8c91"} Sep 17 01:38:03.921606 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.921025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:38:03.922861 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.922830 2582 generic.go:356] "Generic (PLEG): container finished" podID="bde4c47b-878a-400d-a28b-4eb30e925a9f" containerID="90f5e42d942ee9164efe602f2c59ddfc5b4c00ea3f58d16b9c8cfac48dbfb992" exitCode=0 Sep 17 01:38:03.922947 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.922915 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerDied","Data":"90f5e42d942ee9164efe602f2c59ddfc5b4c00ea3f58d16b9c8cfac48dbfb992"} Sep 17 01:38:03.935288 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.935267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:38:03.952793 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:03.952024 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" podStartSLOduration=9.960394247 podStartE2EDuration="27.952002779s" totalImagesPullingTime="17.991608532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:03.951643842 +0000 UTC m=+28.786696168" watchObservedRunningTime="2026-09-17 01:38:03.952002779 +0000 UTC m=+28.787055113" Sep 17 01:38:04.673935 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.673858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:04.674083 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:04.673993 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:04.674393 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.674369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:04.674512 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:04.674492 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:04.927423 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.927204 2582 generic.go:356] "Generic (PLEG): container finished" podID="bde4c47b-878a-400d-a28b-4eb30e925a9f" containerID="610661ec62b9a85cc15ef8a43b423d1b8169a638e867e5a24eee668d92fdbefc" exitCode=0 Sep 17 01:38:04.927746 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.927304 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerDied","Data":"610661ec62b9a85cc15ef8a43b423d1b8169a638e867e5a24eee668d92fdbefc"} Sep 17 01:38:04.928116 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.928089 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:38:04.928258 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.928158 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:38:04.942350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:04.942332 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:38:05.299358 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.299315 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kjf7c"] Sep 17 01:38:05.299482 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.299438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:05.299539 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:05.299526 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:38:05.306302 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.306275 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26n64"] Sep 17 01:38:05.306416 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.306387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:05.306476 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:05.306456 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:05.306833 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.306815 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vkjcl"] Sep 17 01:38:05.306909 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.306894 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:05.307003 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:05.306986 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:05.932097 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.932037 2582 generic.go:356] "Generic (PLEG): container finished" podID="bde4c47b-878a-400d-a28b-4eb30e925a9f" containerID="2640a623b513bc9cd31a60784bb3c26ce443b93771a7ef00b209d24fd0be7aaa" exitCode=0 Sep 17 01:38:05.932448 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:05.932115 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerDied","Data":"2640a623b513bc9cd31a60784bb3c26ce443b93771a7ef00b209d24fd0be7aaa"} Sep 17 01:38:06.673311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:06.673285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:06.673472 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:06.673408 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:06.673472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:06.673454 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:06.673601 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:06.673536 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:07.673359 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:07.673329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:07.673784 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:07.673432 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:38:08.673087 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:08.673055 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:08.673290 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:08.673173 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:08.673290 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:08.673236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:08.673402 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:08.673335 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:08.930855 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:08.930763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:08.983261 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:08.983228 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:09.673941 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:09.673861 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:09.674364 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:09.673982 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjf7c" podUID="cbb95db5-3662-41bf-873e-e2e2e34947fc" Sep 17 01:38:10.673788 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:10.673757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:10.673941 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:10.673760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:10.673941 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:10.673844 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26n64" podUID="f3521de3-ebd7-408e-a2bc-2f03f660b216" Sep 17 01:38:10.673941 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:10.673927 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:38:11.399473 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.399312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:11.399557 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.399498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:11.399557 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.399530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:11.399557 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399451 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:38:11.399707 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399606 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:38:11.399707 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399614 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:43.399590646 +0000 UTC m=+68.234642973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:38:11.399707 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399636 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:38:11.399707 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399659 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:38:11.399707 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399670 2582 projected.go:196] Error preparing data for projected volume kube-api-access-sl8h5 for pod openshift-network-diagnostics/network-check-target-26n64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:38:11.399707 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399642 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret podName:cbb95db5-3662-41bf-873e-e2e2e34947fc nodeName:}" failed. No retries permitted until 2026-09-17 01:38:43.399631542 +0000 UTC m=+68.234683853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret") pod "global-pull-secret-syncer-kjf7c" (UID: "cbb95db5-3662-41bf-873e-e2e2e34947fc") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:38:11.399984 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.399718 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5 podName:f3521de3-ebd7-408e-a2bc-2f03f660b216 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:43.399708868 +0000 UTC m=+68.234761179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sl8h5" (UniqueName: "kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5") pod "network-check-target-26n64" (UID: "f3521de3-ebd7-408e-a2bc-2f03f660b216") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:38:11.435628 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.435607 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeReady" Sep 17 01:38:11.435729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.435705 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 01:38:11.477269 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.477200 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q2l8r"] Sep 17 01:38:11.491695 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.491676 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sd79z"] Sep 17 01:38:11.491823 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.491809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.494532 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.494439 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 01:38:11.495084 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.495061 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 01:38:11.495186 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.495079 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crsck\"" Sep 17 01:38:11.500475 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.500458 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q2l8r"] Sep 17 01:38:11.500560 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.500494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sd79z"] Sep 17 01:38:11.501276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.501259 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.503660 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.503643 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gfnxn\"" Sep 17 01:38:11.503753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.503712 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 01:38:11.503753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.503738 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 01:38:11.503753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.503749 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 01:38:11.600875 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.600855 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-tmp-dir\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.600960 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.600876 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68b2a83b-9d41-4051-a06c-797d0e57edd2-cert\") pod \"ingress-canary-sd79z\" (UID: \"68b2a83b-9d41-4051-a06c-797d0e57edd2\") " pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.600960 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.600909 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-config-volume\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.600960 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.600926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsgt\" (UniqueName: \"kubernetes.io/projected/68b2a83b-9d41-4051-a06c-797d0e57edd2-kube-api-access-7jsgt\") pod \"ingress-canary-sd79z\" (UID: \"68b2a83b-9d41-4051-a06c-797d0e57edd2\") " pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.601075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.601025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.601075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.601061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzcx\" (UniqueName: \"kubernetes.io/projected/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-kube-api-access-kbzcx\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.602519 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.602497 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:11.673417 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.673395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:11.675723 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.675705 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 01:38:11.701363 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzcx\" (UniqueName: \"kubernetes.io/projected/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-kube-api-access-kbzcx\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.701447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-tmp-dir\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.701447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68b2a83b-9d41-4051-a06c-797d0e57edd2-cert\") pod \"ingress-canary-sd79z\" (UID: \"68b2a83b-9d41-4051-a06c-797d0e57edd2\") " pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.701447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-config-volume\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.701618 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsgt\" (UniqueName: \"kubernetes.io/projected/68b2a83b-9d41-4051-a06c-797d0e57edd2-kube-api-access-7jsgt\") pod \"ingress-canary-sd79z\" (UID: \"68b2a83b-9d41-4051-a06c-797d0e57edd2\") " pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.701618 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.701717 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.701616 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:11.701717 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:11.701663 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:12.201648718 +0000 UTC m=+37.036701049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:11.701813 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701729 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-tmp-dir\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.701981 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.701964 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-config-volume\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.705466 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.705452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68b2a83b-9d41-4051-a06c-797d0e57edd2-cert\") pod \"ingress-canary-sd79z\" (UID: \"68b2a83b-9d41-4051-a06c-797d0e57edd2\") " pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.707198 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.707169 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:11.709455 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.709434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzcx\" (UniqueName: \"kubernetes.io/projected/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-kube-api-access-kbzcx\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:11.709824 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.709806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsgt\" (UniqueName: \"kubernetes.io/projected/68b2a83b-9d41-4051-a06c-797d0e57edd2-kube-api-access-7jsgt\") pod \"ingress-canary-sd79z\" (UID: \"68b2a83b-9d41-4051-a06c-797d0e57edd2\") " pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.809922 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.809879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sd79z" Sep 17 01:38:11.951496 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.951465 2582 generic.go:356] "Generic (PLEG): container finished" podID="bde4c47b-878a-400d-a28b-4eb30e925a9f" containerID="6bfc555bdbb44baaf9240ba6ed1bc858f3b747dc17c346f509d8e389d16bf694" exitCode=0 Sep 17 01:38:11.951604 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:11.951528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerDied","Data":"6bfc555bdbb44baaf9240ba6ed1bc858f3b747dc17c346f509d8e389d16bf694"} Sep 17 01:38:12.118803 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.118741 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sd79z"] Sep 17 01:38:12.136006 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:38:12.135978 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b2a83b_9d41_4051_a06c_797d0e57edd2.slice/crio-4490ef451a2a18a8bd72048db4270111fe02110841039c1a309f2da06e40f408 WatchSource:0}: Error finding container 4490ef451a2a18a8bd72048db4270111fe02110841039c1a309f2da06e40f408: Status 404 returned error can't find the container with id 4490ef451a2a18a8bd72048db4270111fe02110841039c1a309f2da06e40f408 Sep 17 01:38:12.205752 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.205728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:12.205879 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:12.205861 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:12.205941 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:12.205932 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:13.205915831 +0000 UTC m=+38.040968142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:12.673633 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.673594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:12.673633 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.673630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:12.676333 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.676309 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 01:38:12.676990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.676321 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 01:38:12.676990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.676342 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fdmc\"" Sep 17 01:38:12.676990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.676373 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 01:38:12.677116 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.676989 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdsht\"" Sep 17 01:38:12.957166 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.956889 2582 generic.go:356] "Generic (PLEG): container finished" podID="bde4c47b-878a-400d-a28b-4eb30e925a9f" containerID="af0cf78965dce23f7c36235fa3726c04b069f744e32d568dfce4e72f2eddcb65" exitCode=0 Sep 17 01:38:12.957325 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.956971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerDied","Data":"af0cf78965dce23f7c36235fa3726c04b069f744e32d568dfce4e72f2eddcb65"} Sep 17 01:38:12.958421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:12.958399 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sd79z" event={"ID":"68b2a83b-9d41-4051-a06c-797d0e57edd2","Type":"ContainerStarted","Data":"4490ef451a2a18a8bd72048db4270111fe02110841039c1a309f2da06e40f408"} Sep 17 01:38:13.213388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:13.213358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:13.213563 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:13.213530 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:13.213682 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:13.213608 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:15.21358781 +0000 UTC m=+40.048640137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:13.965335 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:13.965306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" event={"ID":"bde4c47b-878a-400d-a28b-4eb30e925a9f","Type":"ContainerStarted","Data":"dbc0344ffb787decef58cda3eceb4311a7e3c0765e0b88090e7836b589060b9f"} Sep 17 01:38:13.988088 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:13.986107 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xhrrm" podStartSLOduration=8.789574928 podStartE2EDuration="37.986088608s" totalImagesPullingTime="29.19651368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:13.984719318 +0000 UTC m=+38.819771652" watchObservedRunningTime="2026-09-17 01:38:13.986088608 +0000 UTC m=+38.821140941" Sep 17 01:38:14.103414 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:14.103378 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:14.156309 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:14.156277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:15.228328 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:15.228294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:15.228776 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:15.228434 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:15.228776 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:15.228506 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:19.228487535 +0000 UTC m=+44.063539860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:15.808625 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:15.808592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:15.858419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:15.858391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:15.971528 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:15.971504 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sd79z" event={"ID":"68b2a83b-9d41-4051-a06c-797d0e57edd2","Type":"ContainerStarted","Data":"bec21e67c41ee0b36e568bd521f193969c7596514c5b6036909cc10d7e44ae15"} Sep 17 01:38:15.986356 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:15.986309 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sd79z" podStartSLOduration=2.147721634 podStartE2EDuration="4.986297373s" totalImagesPullingTime="2.838575739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:15.98509808 +0000 UTC m=+40.820150414" watchObservedRunningTime="2026-09-17 01:38:15.986297373 +0000 UTC m=+40.821349705" Sep 17 01:38:19.163248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:19.163192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:19.216761 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:19.216724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:19.252757 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:19.252727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:19.252890 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:19.252868 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:19.252946 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:19.252934 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:27.252918697 +0000 UTC m=+52.087971064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:21.783120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:21.783074 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:21.836653 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:21.836614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:27.306092 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:27.306056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:27.306755 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:27.306260 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:27.306755 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:27.306366 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:38:43.306340784 +0000 UTC m=+68.141393148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:36.945912 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:36.945871 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xzmdf" Sep 17 01:38:37.060709 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:37.060679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:37.162410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:37.162381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:37.217072 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:37.217045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:38:43.314199 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.314166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:38:43.314587 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:43.314283 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:38:43.314587 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:43.314349 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:15.314334178 +0000 UTC m=+100.149386489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:38:43.415142 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.415106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:43.415307 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.415156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:38:43.415307 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.415185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:43.418385 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.418361 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 01:38:43.418522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.418452 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 01:38:43.418522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.418479 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 01:38:43.425754 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:43.425726 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:38:43.425873 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:38:43.425791 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:47.425777722 +0000 UTC m=+132.260830033 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : secret "metrics-daemon-secret" not found Sep 17 01:38:43.427778 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.427751 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 01:38:43.429260 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.429240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbb95db5-3662-41bf-873e-e2e2e34947fc-original-pull-secret\") pod \"global-pull-secret-syncer-kjf7c\" (UID: \"cbb95db5-3662-41bf-873e-e2e2e34947fc\") " pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:43.438696 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.438664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8h5\" (UniqueName: \"kubernetes.io/projected/f3521de3-ebd7-408e-a2bc-2f03f660b216-kube-api-access-sl8h5\") pod \"network-check-target-26n64\" (UID: \"f3521de3-ebd7-408e-a2bc-2f03f660b216\") " pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:43.481118 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.481097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjf7c" Sep 17 01:38:43.594133 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.594069 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdsht\"" Sep 17 01:38:43.601981 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.601960 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:43.747802 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.747779 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kjf7c"] Sep 17 01:38:43.750758 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:38:43.750732 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb95db5_3662_41bf_873e_e2e2e34947fc.slice/crio-25aeb7154f3bf5fd690b6bedb5a2e06e6bc9cfa47e89e9d5c32db6f6ffd6240e WatchSource:0}: Error finding container 25aeb7154f3bf5fd690b6bedb5a2e06e6bc9cfa47e89e9d5c32db6f6ffd6240e: Status 404 returned error can't find the container with id 25aeb7154f3bf5fd690b6bedb5a2e06e6bc9cfa47e89e9d5c32db6f6ffd6240e Sep 17 01:38:43.863078 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:43.863025 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26n64"] Sep 17 01:38:43.865449 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:38:43.865431 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3521de3_ebd7_408e_a2bc_2f03f660b216.slice/crio-0dc1616c4d05f2da9446336a50f2203f236ecd273261d070b7dab7dde32be646 WatchSource:0}: Error finding container 0dc1616c4d05f2da9446336a50f2203f236ecd273261d070b7dab7dde32be646: Status 404 returned error can't find the container with id 0dc1616c4d05f2da9446336a50f2203f236ecd273261d070b7dab7dde32be646 Sep 17 01:38:44.034005 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:44.033977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26n64" event={"ID":"f3521de3-ebd7-408e-a2bc-2f03f660b216","Type":"ContainerStarted","Data":"0dc1616c4d05f2da9446336a50f2203f236ecd273261d070b7dab7dde32be646"} Sep 17 01:38:44.035232 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:44.035196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kjf7c" event={"ID":"cbb95db5-3662-41bf-873e-e2e2e34947fc","Type":"ContainerStarted","Data":"25aeb7154f3bf5fd690b6bedb5a2e06e6bc9cfa47e89e9d5c32db6f6ffd6240e"} Sep 17 01:38:49.049967 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:49.049932 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26n64" event={"ID":"f3521de3-ebd7-408e-a2bc-2f03f660b216","Type":"ContainerStarted","Data":"1b05a99eea63c61b99abc2d3b64b146344ad071a02d4344bb86228c5040ddaf1"} Sep 17 01:38:49.050431 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:49.050061 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:38:49.051341 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:49.051316 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kjf7c" event={"ID":"cbb95db5-3662-41bf-873e-e2e2e34947fc","Type":"ContainerStarted","Data":"cef4bfb08049da0754a552ce73f02d57180d82d8d2b7d211fe26a728c483663a"} Sep 17 01:38:49.065689 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:49.065647 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-26n64" podStartSLOduration=68.713587968 podStartE2EDuration="1m13.065632294s" totalImagesPullingTime="4.352044326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:49.065130077 +0000 UTC m=+73.900182410" watchObservedRunningTime="2026-09-17 01:38:49.065632294 +0000 UTC m=+73.900684623" Sep 17 01:38:49.078268 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:49.078203 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kjf7c" podStartSLOduration=68.608544356 podStartE2EDuration="1m13.078187382s" totalImagesPullingTime="4.469643026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:49.07799738 +0000 UTC m=+73.913049714" watchObservedRunningTime="2026-09-17 01:38:49.078187382 +0000 UTC m=+73.913239713" Sep 17 01:38:59.406611 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.406492 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l"] Sep 17 01:38:59.410378 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.410360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.412539 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.412502 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg"] Sep 17 01:38:59.413617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.413446 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 01:38:59.413617 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.413590 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 01:38:59.413817 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.413592 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 17 01:38:59.414540 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.414520 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 01:38:59.415519 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.415497 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l"] Sep 17 01:38:59.415635 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.415580 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.417763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.417742 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 17 01:38:59.417858 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.417766 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xvxh8\"" Sep 17 01:38:59.425393 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.425372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg"] Sep 17 01:38:59.515638 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.515616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5l2\" (UniqueName: \"kubernetes.io/projected/a7d22895-1632-443d-b4df-c0ea2fa8a197-kube-api-access-ll5l2\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.515726 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.515641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7d22895-1632-443d-b4df-c0ea2fa8a197-klusterlet-config\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.515726 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.515672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7077992-404f-4992-bc79-5b0415ae697c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6579594d98-msfmg\" (UID: \"a7077992-404f-4992-bc79-5b0415ae697c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.515815 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.515722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ps9\" (UniqueName: \"kubernetes.io/projected/a7077992-404f-4992-bc79-5b0415ae697c-kube-api-access-t4ps9\") pod \"managed-serviceaccount-addon-agent-6579594d98-msfmg\" (UID: \"a7077992-404f-4992-bc79-5b0415ae697c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.515815 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.515751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d22895-1632-443d-b4df-c0ea2fa8a197-tmp\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.616521 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.616490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ps9\" (UniqueName: \"kubernetes.io/projected/a7077992-404f-4992-bc79-5b0415ae697c-kube-api-access-t4ps9\") pod \"managed-serviceaccount-addon-agent-6579594d98-msfmg\" (UID: \"a7077992-404f-4992-bc79-5b0415ae697c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.616594 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.616526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d22895-1632-443d-b4df-c0ea2fa8a197-tmp\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.616728 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.616710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5l2\" (UniqueName: \"kubernetes.io/projected/a7d22895-1632-443d-b4df-c0ea2fa8a197-kube-api-access-ll5l2\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.616772 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.616742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7d22895-1632-443d-b4df-c0ea2fa8a197-klusterlet-config\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.616807 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.616780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7077992-404f-4992-bc79-5b0415ae697c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6579594d98-msfmg\" (UID: \"a7077992-404f-4992-bc79-5b0415ae697c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.616903 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.616885 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d22895-1632-443d-b4df-c0ea2fa8a197-tmp\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.619543 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.619521 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a7d22895-1632-443d-b4df-c0ea2fa8a197-klusterlet-config\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.619693 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.619669 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7077992-404f-4992-bc79-5b0415ae697c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6579594d98-msfmg\" (UID: \"a7077992-404f-4992-bc79-5b0415ae697c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.624113 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.624088 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5l2\" (UniqueName: \"kubernetes.io/projected/a7d22895-1632-443d-b4df-c0ea2fa8a197-kube-api-access-ll5l2\") pod \"klusterlet-addon-workmgr-79b59949f7-hvp5l\" (UID: \"a7d22895-1632-443d-b4df-c0ea2fa8a197\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.624539 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.624515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ps9\" (UniqueName: \"kubernetes.io/projected/a7077992-404f-4992-bc79-5b0415ae697c-kube-api-access-t4ps9\") pod \"managed-serviceaccount-addon-agent-6579594d98-msfmg\" (UID: \"a7077992-404f-4992-bc79-5b0415ae697c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.722683 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.722654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:38:59.742689 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.742666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" Sep 17 01:38:59.994723 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:38:59.994660 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l"] Sep 17 01:38:59.998132 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:38:59.998106 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d22895_1632_443d_b4df_c0ea2fa8a197.slice/crio-b3114394e28113662f93ddccf4b7974b0ff66658cea6b9ead8e06b416a7d0d25 WatchSource:0}: Error finding container b3114394e28113662f93ddccf4b7974b0ff66658cea6b9ead8e06b416a7d0d25: Status 404 returned error can't find the container with id b3114394e28113662f93ddccf4b7974b0ff66658cea6b9ead8e06b416a7d0d25 Sep 17 01:39:00.017516 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:00.017494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg"] Sep 17 01:39:00.021352 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:39:00.021327 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7077992_404f_4992_bc79_5b0415ae697c.slice/crio-2b2f5f522ab930b1cea35bea6bccc47125622773a945face8419f3b66f980e83 WatchSource:0}: Error finding container 2b2f5f522ab930b1cea35bea6bccc47125622773a945face8419f3b66f980e83: Status 404 returned error can't find the container with id 2b2f5f522ab930b1cea35bea6bccc47125622773a945face8419f3b66f980e83 Sep 17 01:39:00.081581 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:00.081548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" event={"ID":"a7077992-404f-4992-bc79-5b0415ae697c","Type":"ContainerStarted","Data":"2b2f5f522ab930b1cea35bea6bccc47125622773a945face8419f3b66f980e83"} Sep 17 01:39:00.082636 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:00.082618 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" event={"ID":"a7d22895-1632-443d-b4df-c0ea2fa8a197","Type":"ContainerStarted","Data":"b3114394e28113662f93ddccf4b7974b0ff66658cea6b9ead8e06b416a7d0d25"} Sep 17 01:39:04.097027 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:04.096994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" event={"ID":"a7077992-404f-4992-bc79-5b0415ae697c","Type":"ContainerStarted","Data":"b2f93548f9de92fde9d29250769c6e0fc324170a1692bd274eafae4f74a25b8c"} Sep 17 01:39:04.110866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:04.110808 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" podStartSLOduration=1.6879550349999999 podStartE2EDuration="5.110794469s" totalImagesPullingTime="3.422839434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:04.110227615 +0000 UTC m=+88.945279942" watchObservedRunningTime="2026-09-17 01:39:04.110794469 +0000 UTC m=+88.945846803" Sep 17 01:39:05.100990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:05.100953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" event={"ID":"a7d22895-1632-443d-b4df-c0ea2fa8a197","Type":"ContainerStarted","Data":"727e6563bdd6fee3ab2fc91e91bb26f81b78c2b2f92f12a11c81b0e60565b3a9"} Sep 17 01:39:05.115964 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:05.115920 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" podStartSLOduration=1.471027199 podStartE2EDuration="6.115904987s" totalImagesPullingTime="4.644877788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:05.115441622 +0000 UTC m=+89.950493955" watchObservedRunningTime="2026-09-17 01:39:05.115904987 +0000 UTC m=+89.950957319" Sep 17 01:39:06.103630 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:06.103589 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:39:06.105166 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:06.105142 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:39:15.315102 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:15.315069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:39:15.315449 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:39:15.315226 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:39:15.315449 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:39:15.315295 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:40:19.315280671 +0000 UTC m=+164.150333007 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:39:20.055969 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:20.055933 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-26n64" Sep 17 01:39:23.589165 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:23.589125 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:39:47.513337 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:39:47.513285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:39:47.513872 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:39:47.513412 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:39:47.513872 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:39:47.513514 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs podName:bf91a16c-6494-4e30-a6e2-1cdefa9856c6 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:49.513495483 +0000 UTC m=+254.348547805 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs") pod "network-metrics-daemon-vkjcl" (UID: "bf91a16c-6494-4e30-a6e2-1cdefa9856c6") : secret "metrics-daemon-secret" not found Sep 17 01:40:14.503204 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:40:14.503149 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q2l8r" podUID="cdcd7a2a-0e54-4fef-abb6-a1046f64fa21" Sep 17 01:40:15.272937 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:40:15.272908 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q2l8r" Sep 17 01:40:15.685257 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:40:15.685178 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vkjcl" podUID="bf91a16c-6494-4e30-a6e2-1cdefa9856c6" Sep 17 01:40:19.411698 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:40:19.411659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:40:19.412235 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:40:19.411804 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:40:19.412235 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:40:19.411871 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls podName:cdcd7a2a-0e54-4fef-abb6-a1046f64fa21 nodeName:}" failed. No retries permitted until 2026-09-17 01:42:21.411851838 +0000 UTC m=+286.246904175 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls") pod "dns-default-q2l8r" (UID: "cdcd7a2a-0e54-4fef-abb6-a1046f64fa21") : secret "dns-default-metrics-tls" not found Sep 17 01:40:28.673202 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:40:28.673134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:41:01.830688 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:01.830651 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:04.390283 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:04.390246 2582 generic.go:356] "Generic (PLEG): container finished" podID="a7077992-404f-4992-bc79-5b0415ae697c" containerID="b2f93548f9de92fde9d29250769c6e0fc324170a1692bd274eafae4f74a25b8c" exitCode=255 Sep 17 01:41:04.390650 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:04.390292 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" event={"ID":"a7077992-404f-4992-bc79-5b0415ae697c","Type":"ContainerDied","Data":"b2f93548f9de92fde9d29250769c6e0fc324170a1692bd274eafae4f74a25b8c"} Sep 17 01:41:04.396152 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:04.396131 2582 scope.go:117] "RemoveContainer" containerID="b2f93548f9de92fde9d29250769c6e0fc324170a1692bd274eafae4f74a25b8c" Sep 17 01:41:05.397550 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:05.397515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6579594d98-msfmg" event={"ID":"a7077992-404f-4992-bc79-5b0415ae697c","Type":"ContainerStarted","Data":"be4a577eac0b46a1c5b3dcc3e721ab00466bcaa167812cbde96e0cb2218edcae"} Sep 17 01:41:05.399139 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:05.399112 2582 generic.go:356] "Generic (PLEG): container finished" podID="a7d22895-1632-443d-b4df-c0ea2fa8a197" containerID="727e6563bdd6fee3ab2fc91e91bb26f81b78c2b2f92f12a11c81b0e60565b3a9" exitCode=1 Sep 17 01:41:05.399299 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:05.399183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" event={"ID":"a7d22895-1632-443d-b4df-c0ea2fa8a197","Type":"ContainerDied","Data":"727e6563bdd6fee3ab2fc91e91bb26f81b78c2b2f92f12a11c81b0e60565b3a9"} Sep 17 01:41:05.399507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:05.399493 2582 scope.go:117] "RemoveContainer" containerID="727e6563bdd6fee3ab2fc91e91bb26f81b78c2b2f92f12a11c81b0e60565b3a9" Sep 17 01:41:06.104372 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:06.104330 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:41:06.404036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:06.403946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" event={"ID":"a7d22895-1632-443d-b4df-c0ea2fa8a197","Type":"ContainerStarted","Data":"0c9619af613bcaac908cb2f0baff0f61f6d26350c565c039dc18d5ee45f0f620"} Sep 17 01:41:06.404459 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:06.404173 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:41:06.404813 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:06.404795 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79b59949f7-hvp5l" Sep 17 01:41:22.789260 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.789227 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ltc72"] Sep 17 01:41:22.791237 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.791201 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" Sep 17 01:41:22.794328 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.794307 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9vv2f\"" Sep 17 01:41:22.805541 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.805518 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ltc72"] Sep 17 01:41:22.821014 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.820991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68drb\" (UniqueName: \"kubernetes.io/projected/980fd693-bede-44a4-9158-a3bef2971be9-kube-api-access-68drb\") pod \"network-check-source-6f8b768455-ltc72\" (UID: \"980fd693-bede-44a4-9158-a3bef2971be9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" Sep 17 01:41:22.887050 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.887015 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp"] Sep 17 01:41:22.888957 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.888936 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:22.890713 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.890689 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g"] Sep 17 01:41:22.891447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.891432 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ncxqh\"" Sep 17 01:41:22.891533 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.891435 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.891665 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.891649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 01:41:22.891711 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.891690 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.892423 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.892406 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d848974c4-prkmc"] Sep 17 01:41:22.892529 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.892514 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:22.893992 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.893976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.894910 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.894893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x6rr2\"" Sep 17 01:41:22.895155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.895133 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 01:41:22.895409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.895393 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 01:41:22.896141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.896123 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 01:41:22.896255 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.896243 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 01:41:22.896350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.896334 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7d5d6\"" Sep 17 01:41:22.896951 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.896921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 01:41:22.896951 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.896921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 01:41:22.897967 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.897944 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:41:22.901895 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.901875 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 01:41:22.902120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.902101 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp"] Sep 17 01:41:22.909555 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.909531 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g"] Sep 17 01:41:22.922021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.921997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68drb\" (UniqueName: \"kubernetes.io/projected/980fd693-bede-44a4-9158-a3bef2971be9-kube-api-access-68drb\") pod \"network-check-source-6f8b768455-ltc72\" (UID: \"980fd693-bede-44a4-9158-a3bef2971be9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" Sep 17 01:41:22.922120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922043 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-ca-trust-extracted\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:22.922120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-installation-pull-secrets\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-bound-sa-token\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-trusted-ca\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45rr\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-kube-api-access-t45rr\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad2fb6c-1185-4211-8e82-ae810c248e8b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:22.922502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95pw\" (UniqueName: \"kubernetes.io/projected/5ad2fb6c-1185-4211-8e82-ae810c248e8b-kube-api-access-s95pw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:22.922502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-image-registry-private-configuration\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-certificates\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:22.922502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlk4k\" (UniqueName: \"kubernetes.io/projected/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-kube-api-access-dlk4k\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:22.922502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.922430 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ad2fb6c-1185-4211-8e82-ae810c248e8b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:22.926884 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.926862 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d848974c4-prkmc"] Sep 17 01:41:22.933277 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:22.933258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68drb\" (UniqueName: \"kubernetes.io/projected/980fd693-bede-44a4-9158-a3bef2971be9-kube-api-access-68drb\") pod \"network-check-source-6f8b768455-ltc72\" (UID: \"980fd693-bede-44a4-9158-a3bef2971be9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" Sep 17 01:41:23.015552 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.015525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:23.023151 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlk4k\" (UniqueName: \"kubernetes.io/projected/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-kube-api-access-dlk4k\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:23.023255 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ad2fb6c-1185-4211-8e82-ae810c248e8b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.023255 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023359 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.023283 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:41:23.023359 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.023295 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d848974c4-prkmc: secret "image-registry-tls" not found Sep 17 01:41:23.023359 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023325 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-ca-trust-extracted\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023359 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.023342 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls podName:d7cb31a9-2ccc-457e-9b48-60e7d5d92371 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:23.523328528 +0000 UTC m=+228.358380840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls") pod "image-registry-5d848974c4-prkmc" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371") : secret "image-registry-tls" not found Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-installation-pull-secrets\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-bound-sa-token\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-trusted-ca\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45rr\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-kube-api-access-t45rr\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.023480 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad2fb6c-1185-4211-8e82-ae810c248e8b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.023559 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.023530 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls podName:bee6c20e-0c73-4d6b-9fc5-a559867f2c72 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:23.523514148 +0000 UTC m=+228.358566483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-hm5jp" (UID: "bee6c20e-0c73-4d6b-9fc5-a559867f2c72") : secret "samples-operator-tls" not found Sep 17 01:41:23.023954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95pw\" (UniqueName: \"kubernetes.io/projected/5ad2fb6c-1185-4211-8e82-ae810c248e8b-kube-api-access-s95pw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.023954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-image-registry-private-configuration\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-certificates\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.023954 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ad2fb6c-1185-4211-8e82-ae810c248e8b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.024106 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.023995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-ca-trust-extracted\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.024338 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.024318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-certificates\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.024518 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.024495 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-trusted-ca\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.025810 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.025790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-installation-pull-secrets\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.025885 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.025824 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ad2fb6c-1185-4211-8e82-ae810c248e8b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.025935 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.025912 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-image-registry-private-configuration\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.033082 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.033049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95pw\" (UniqueName: \"kubernetes.io/projected/5ad2fb6c-1185-4211-8e82-ae810c248e8b-kube-api-access-s95pw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7hm4g\" (UID: \"5ad2fb6c-1185-4211-8e82-ae810c248e8b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.033411 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.033394 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlk4k\" (UniqueName: \"kubernetes.io/projected/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-kube-api-access-dlk4k\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:23.033481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.033426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45rr\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-kube-api-access-t45rr\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.033904 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.033888 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-bound-sa-token\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.099447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.099369 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" Sep 17 01:41:23.206876 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.206839 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" Sep 17 01:41:23.362833 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.362769 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ltc72"] Sep 17 01:41:23.365703 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:41:23.365679 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980fd693_bede_44a4_9158_a3bef2971be9.slice/crio-95df378633284847fa79aeae20d5022b33528542803d3a385972ae10c64212cc WatchSource:0}: Error finding container 95df378633284847fa79aeae20d5022b33528542803d3a385972ae10c64212cc: Status 404 returned error can't find the container with id 95df378633284847fa79aeae20d5022b33528542803d3a385972ae10c64212cc Sep 17 01:41:23.456020 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.455970 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" event={"ID":"980fd693-bede-44a4-9158-a3bef2971be9","Type":"ContainerStarted","Data":"512e9c611fbab835e95edd48260ca6bc2f21940fd32494d519120425713589da"} Sep 17 01:41:23.456185 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.456029 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" event={"ID":"980fd693-bede-44a4-9158-a3bef2971be9","Type":"ContainerStarted","Data":"95df378633284847fa79aeae20d5022b33528542803d3a385972ae10c64212cc"} Sep 17 01:41:23.469814 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.469769 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ltc72" podStartSLOduration=1.469754144 podStartE2EDuration="1.469754144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:23.469204447 +0000 UTC m=+228.304256790" watchObservedRunningTime="2026-09-17 01:41:23.469754144 +0000 UTC m=+228.304806477" Sep 17 01:41:23.473669 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.473642 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g"] Sep 17 01:41:23.476320 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:41:23.476291 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad2fb6c_1185_4211_8e82_ae810c248e8b.slice/crio-760b48cee6c1f0997bee337e04df553e1897cfc7fa44b05d5beb62dfe0c2c880 WatchSource:0}: Error finding container 760b48cee6c1f0997bee337e04df553e1897cfc7fa44b05d5beb62dfe0c2c880: Status 404 returned error can't find the container with id 760b48cee6c1f0997bee337e04df553e1897cfc7fa44b05d5beb62dfe0c2c880 Sep 17 01:41:23.527495 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.527453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:23.527495 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:23.527495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:23.527743 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.527588 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 01:41:23.527743 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.527607 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:41:23.527743 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.527626 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d848974c4-prkmc: secret "image-registry-tls" not found Sep 17 01:41:23.527743 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.527650 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls podName:bee6c20e-0c73-4d6b-9fc5-a559867f2c72 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:24.527630727 +0000 UTC m=+229.362683052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-hm5jp" (UID: "bee6c20e-0c73-4d6b-9fc5-a559867f2c72") : secret "samples-operator-tls" not found Sep 17 01:41:23.527743 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:23.527678 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls podName:d7cb31a9-2ccc-457e-9b48-60e7d5d92371 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:24.52766193 +0000 UTC m=+229.362714242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls") pod "image-registry-5d848974c4-prkmc" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371") : secret "image-registry-tls" not found Sep 17 01:41:24.274987 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:24.274935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:24.461664 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:24.461547 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" event={"ID":"5ad2fb6c-1185-4211-8e82-ae810c248e8b","Type":"ContainerStarted","Data":"760b48cee6c1f0997bee337e04df553e1897cfc7fa44b05d5beb62dfe0c2c880"} Sep 17 01:41:24.534536 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:24.534459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:24.534536 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:24.534494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:24.534730 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:24.534588 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 01:41:24.534730 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:24.534609 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:41:24.534730 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:24.534626 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d848974c4-prkmc: secret "image-registry-tls" not found Sep 17 01:41:24.534730 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:24.534651 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls podName:bee6c20e-0c73-4d6b-9fc5-a559867f2c72 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:26.534633461 +0000 UTC m=+231.369685775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-hm5jp" (UID: "bee6c20e-0c73-4d6b-9fc5-a559867f2c72") : secret "samples-operator-tls" not found Sep 17 01:41:24.534730 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:24.534671 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls podName:d7cb31a9-2ccc-457e-9b48-60e7d5d92371 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:26.534660408 +0000 UTC m=+231.369712718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls") pod "image-registry-5d848974c4-prkmc" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371") : secret "image-registry-tls" not found Sep 17 01:41:25.466287 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:25.466251 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" event={"ID":"5ad2fb6c-1185-4211-8e82-ae810c248e8b","Type":"ContainerStarted","Data":"a0fc23f99e20b2057f86d533526d7a521e8fbe8c0f90f32430d19da85fcdfa63"} Sep 17 01:41:25.481351 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:25.481310 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" podStartSLOduration=1.7894978419999998 podStartE2EDuration="3.48129723s" totalImagesPullingTime="1.691799388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:25.480338605 +0000 UTC m=+230.315390939" watchObservedRunningTime="2026-09-17 01:41:25.48129723 +0000 UTC m=+230.316349562" Sep 17 01:41:25.623473 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:25.623441 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:25.675268 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:25.675242 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:26.327125 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:26.327090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:26.418679 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:26.418657 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:26.519310 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:26.519285 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:26.550412 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:26.550388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:26.550529 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:26.550420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:26.550529 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:26.550510 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:41:26.550529 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:26.550526 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 01:41:26.550661 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:26.550530 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d848974c4-prkmc: secret "image-registry-tls" not found Sep 17 01:41:26.550661 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:26.550590 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls podName:bee6c20e-0c73-4d6b-9fc5-a559867f2c72 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:30.550571852 +0000 UTC m=+235.385624168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-hm5jp" (UID: "bee6c20e-0c73-4d6b-9fc5-a559867f2c72") : secret "samples-operator-tls" not found Sep 17 01:41:26.550661 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:26.550612 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls podName:d7cb31a9-2ccc-457e-9b48-60e7d5d92371 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:30.550599861 +0000 UTC m=+235.385652175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls") pod "image-registry-5d848974c4-prkmc" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371") : secret "image-registry-tls" not found Sep 17 01:41:27.593765 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.593729 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hn297"] Sep 17 01:41:27.596865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.596843 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.599130 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.599110 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 01:41:27.600108 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.600076 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 01:41:27.600108 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.600079 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 01:41:27.600284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.600081 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 01:41:27.600284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.600152 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4q7v\"" Sep 17 01:41:27.608052 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.608014 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hn297"] Sep 17 01:41:27.618484 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.618464 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:27.658652 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.658623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.658737 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.658665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.658737 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.658699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-crio-socket\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.658818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.658757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-data-volume\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.658818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.658803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8qs\" (UniqueName: \"kubernetes.io/projected/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-api-access-wt8qs\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.672699 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.672669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:27.759158 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.759253 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.759253 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-crio-socket\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.759253 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-data-volume\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.759416 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:27.759272 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:27.759416 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8qs\" (UniqueName: \"kubernetes.io/projected/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-api-access-wt8qs\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.759416 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:27.759339 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:28.259321259 +0000 UTC m=+233.094373570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : configmap "kube-rbac-proxy" not found Sep 17 01:41:27.759416 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-crio-socket\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.759416 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:27.759393 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:27.759608 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:27.759449 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:28.25943586 +0000 UTC m=+233.094488170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:27.759608 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.759537 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-data-volume\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:27.769303 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:27.769283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8qs\" (UniqueName: \"kubernetes.io/projected/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-api-access-wt8qs\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:28.121287 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.121251 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:28.262802 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.262775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:28.262893 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.262809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:28.262929 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:28.262894 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:28.262929 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:28.262909 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:28.263018 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:28.262940 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:29.262928068 +0000 UTC m=+234.097980396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : configmap "kube-rbac-proxy" not found Sep 17 01:41:28.263018 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:28.262987 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:29.262976507 +0000 UTC m=+234.098028819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:28.322730 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.322688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:28.380365 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.380304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:28.557918 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.557891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:28.608405 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.608375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:28.829191 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.829160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:28.930387 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:28.930359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:29.268181 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:29.268149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:29.268324 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:29.268190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:29.268324 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:29.268298 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:29.268393 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:29.268350 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:31.268337316 +0000 UTC m=+236.103389628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:29.268393 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:29.268302 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:29.268393 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:29.268381 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:31.26837469 +0000 UTC m=+236.103427001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : configmap "kube-rbac-proxy" not found Sep 17 01:41:30.158039 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.158013 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-498hn_5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0/dns-node-resolver/0.log" Sep 17 01:41:30.428368 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.428302 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-kg97x"] Sep 17 01:41:30.431302 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.431287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.433855 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.433837 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 17 01:41:30.434900 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.434873 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ntq9n\"" Sep 17 01:41:30.434900 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.434898 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 17 01:41:30.435049 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.434880 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 17 01:41:30.435049 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.434880 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 17 01:41:30.438888 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.438868 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-kg97x"] Sep 17 01:41:30.461946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.461919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:30.477270 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.477234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2f26ddf-6399-499d-9d42-68fd6106d732-signing-cabundle\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.477359 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.477279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2f26ddf-6399-499d-9d42-68fd6106d732-signing-key\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.477359 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.477308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x688l\" (UniqueName: \"kubernetes.io/projected/f2f26ddf-6399-499d-9d42-68fd6106d732-kube-api-access-x688l\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.577639 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.577586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2f26ddf-6399-499d-9d42-68fd6106d732-signing-key\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.577818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.577657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x688l\" (UniqueName: \"kubernetes.io/projected/f2f26ddf-6399-499d-9d42-68fd6106d732-kube-api-access-x688l\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.577818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.577727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:30.577818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.577758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:30.577818 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.577806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2f26ddf-6399-499d-9d42-68fd6106d732-signing-cabundle\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.578005 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:30.577906 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 01:41:30.578005 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:30.577969 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls podName:bee6c20e-0c73-4d6b-9fc5-a559867f2c72 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:38.577950956 +0000 UTC m=+243.413003268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-hm5jp" (UID: "bee6c20e-0c73-4d6b-9fc5-a559867f2c72") : secret "samples-operator-tls" not found Sep 17 01:41:30.578005 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:30.577906 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:41:30.578005 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:30.578002 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d848974c4-prkmc: secret "image-registry-tls" not found Sep 17 01:41:30.578183 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:30.578056 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls podName:d7cb31a9-2ccc-457e-9b48-60e7d5d92371 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:38.578039944 +0000 UTC m=+243.413092279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls") pod "image-registry-5d848974c4-prkmc" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371") : secret "image-registry-tls" not found Sep 17 01:41:30.578419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.578403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2f26ddf-6399-499d-9d42-68fd6106d732-signing-cabundle\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.580043 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.580027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2f26ddf-6399-499d-9d42-68fd6106d732-signing-key\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.585005 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.584985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x688l\" (UniqueName: \"kubernetes.io/projected/f2f26ddf-6399-499d-9d42-68fd6106d732-kube-api-access-x688l\") pod \"service-ca-568869fdc5-kg97x\" (UID: \"f2f26ddf-6399-499d-9d42-68fd6106d732\") " pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:30.740859 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:30.740827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-kg97x" Sep 17 01:41:31.014223 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:31.014144 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-kg97x"] Sep 17 01:41:31.017156 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:41:31.017130 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f26ddf_6399_499d_9d42_68fd6106d732.slice/crio-a33ec5ad46d7f8a1acbf1d08a697637716274c3ab4f6c8d3dbfd7bc5659b992d WatchSource:0}: Error finding container a33ec5ad46d7f8a1acbf1d08a697637716274c3ab4f6c8d3dbfd7bc5659b992d: Status 404 returned error can't find the container with id a33ec5ad46d7f8a1acbf1d08a697637716274c3ab4f6c8d3dbfd7bc5659b992d Sep 17 01:41:31.158929 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:31.158897 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4687l_7c84a533-aeac-4d80-b820-f72b570c337d/node-ca/0.log" Sep 17 01:41:31.283889 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:31.283802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:31.284053 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:31.283901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:31.284053 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:31.283943 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:31.284053 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:31.284005 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:31.284053 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:31.284018 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:35.283999185 +0000 UTC m=+240.119051502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : configmap "kube-rbac-proxy" not found Sep 17 01:41:31.284053 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:31.284044 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:35.284033083 +0000 UTC m=+240.119085395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:31.305052 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:31.305023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:31.483730 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:31.483695 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-kg97x" event={"ID":"f2f26ddf-6399-499d-9d42-68fd6106d732","Type":"ContainerStarted","Data":"a33ec5ad46d7f8a1acbf1d08a697637716274c3ab4f6c8d3dbfd7bc5659b992d"} Sep 17 01:41:32.358792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:32.358753 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sd79z_68b2a83b-9d41-4051-a06c-797d0e57edd2/serve-healthcheck-canary/0.log" Sep 17 01:41:32.959168 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:32.959127 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-7hm4g_5ad2fb6c-1185-4211-8e82-ae810c248e8b/kube-storage-version-migrator-operator/0.log" Sep 17 01:41:33.491886 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:33.491853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-kg97x" event={"ID":"f2f26ddf-6399-499d-9d42-68fd6106d732","Type":"ContainerStarted","Data":"50935a559d3bc0a5f44a6a3c319b23dffe42efe19215e86993cb275c60fe7954"} Sep 17 01:41:33.508123 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:33.508078 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-kg97x" podStartSLOduration=1.6285682750000001 podStartE2EDuration="3.508063783s" totalImagesPullingTime="1.879495508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:33.507566273 +0000 UTC m=+238.342618607" watchObservedRunningTime="2026-09-17 01:41:33.508063783 +0000 UTC m=+238.343116115" Sep 17 01:41:33.775149 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:33.775062 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:34.653587 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:34.653542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:34.960689 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:34.960612 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:35.014585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.014549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:35.217126 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.217093 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:35.268017 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.267968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:35.316469 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.316437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:35.316634 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.316506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:35.316695 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:35.316682 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:41:35.316749 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:35.316741 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:43.31672279 +0000 UTC m=+248.151775114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : configmap "kube-rbac-proxy" not found Sep 17 01:41:35.317146 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:35.317129 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:35.317198 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:35.317178 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:43.31716542 +0000 UTC m=+248.152217735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:35.340882 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.340856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:35.439893 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.439861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:35.954386 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:35.954352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:36.751685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:36.751652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:36.948395 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:36.948359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:37.353875 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:37.353842 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:37.554375 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:37.554341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:37.950422 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:37.950373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:38.123613 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.123567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:38.550575 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.550533 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:38.598295 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.598260 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:38.644488 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.644462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") pod \"image-registry-5d848974c4-prkmc\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:38.644620 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.644495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:38.644661 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:38.644629 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 01:41:38.644661 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:38.644654 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d848974c4-prkmc: secret "image-registry-tls" not found Sep 17 01:41:38.644728 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:38.644722 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls podName:d7cb31a9-2ccc-457e-9b48-60e7d5d92371 nodeName:}" failed. No retries permitted until 2026-09-17 01:41:54.64469932 +0000 UTC m=+259.479751636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls") pod "image-registry-5d848974c4-prkmc" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371") : secret "image-registry-tls" not found Sep 17 01:41:38.646968 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.646949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bee6c20e-0c73-4d6b-9fc5-a559867f2c72-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-hm5jp\" (UID: \"bee6c20e-0c73-4d6b-9fc5-a559867f2c72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:38.804236 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.804131 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ncxqh\"" Sep 17 01:41:38.812218 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.812189 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" Sep 17 01:41:38.948817 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.948775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:38.995830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:38.995800 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:39.083889 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:39.083698 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp"] Sep 17 01:41:39.156363 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:39.156330 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:39.496276 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:39.496242 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:39.511942 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:39.511905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" event={"ID":"bee6c20e-0c73-4d6b-9fc5-a559867f2c72","Type":"ContainerStarted","Data":"e5c0ea96210e85b0cb19f94e7fa4e38759191b587a9782b0d01c899a535cef8a"} Sep 17 01:41:39.596915 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:39.596865 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:39.954141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:39.954018 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:40.752424 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:40.752394 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:41.145916 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.145881 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:41.203375 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.203339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:41.351546 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.351506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:41.521269 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.521231 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" event={"ID":"bee6c20e-0c73-4d6b-9fc5-a559867f2c72","Type":"ContainerStarted","Data":"9b5ec90643875ddadea1045b76c6de4948d7eca2cdfd037603c5b1a650fbf111"} Sep 17 01:41:41.521485 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.521285 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" event={"ID":"bee6c20e-0c73-4d6b-9fc5-a559867f2c72","Type":"ContainerStarted","Data":"03e3396ad39a053f645de3c1db5cf6f137698ddaa61e30495518c61289af3aa0"} Sep 17 01:41:41.537201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.537140 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-hm5jp" podStartSLOduration=17.970050244 podStartE2EDuration="19.537120552s" totalImagesPullingTime="1.567070308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:41.536473958 +0000 UTC m=+246.371526291" watchObservedRunningTime="2026-09-17 01:41:41.537120552 +0000 UTC m=+246.372172885" Sep 17 01:41:41.557047 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.557010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:41.747888 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:41.747853 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:42.752610 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:42.752568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:42.950097 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:42.950045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:43.352057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:43.352024 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:43.384194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:43.384161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:43.384365 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:43.384223 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:43.384365 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:43.384338 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:41:43.384445 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:43.384413 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls podName:f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc nodeName:}" failed. No retries permitted until 2026-09-17 01:41:59.384391849 +0000 UTC m=+264.219444165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn297" (UID: "f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc") : secret "insights-runtime-extractor-tls" not found Sep 17 01:41:43.384914 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:43.384892 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:43.548201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:43.548157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:43.951523 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:43.951491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:44.554184 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:44.554150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:44.945484 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:44.945403 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:45.149403 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:45.149366 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:45.355535 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:45.355504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:46.792850 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:46.792819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:49.533318 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:49.533278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:41:49.535962 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:49.535924 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf91a16c-6494-4e30-a6e2-1cdefa9856c6-metrics-certs\") pod \"network-metrics-daemon-vkjcl\" (UID: \"bf91a16c-6494-4e30-a6e2-1cdefa9856c6\") " pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:41:49.676533 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:49.676508 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fdmc\"" Sep 17 01:41:49.684449 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:49.684428 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkjcl" Sep 17 01:41:49.951649 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:49.951574 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vkjcl"] Sep 17 01:41:49.955892 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:41:49.955865 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf91a16c_6494_4e30_a6e2_1cdefa9856c6.slice/crio-29ca861022bf520c2573e321b72dcc38755daa1eaebe8174caf60200234754e3 WatchSource:0}: Error finding container 29ca861022bf520c2573e321b72dcc38755daa1eaebe8174caf60200234754e3: Status 404 returned error can't find the container with id 29ca861022bf520c2573e321b72dcc38755daa1eaebe8174caf60200234754e3 Sep 17 01:41:50.554164 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:50.554120 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vkjcl" event={"ID":"bf91a16c-6494-4e30-a6e2-1cdefa9856c6","Type":"ContainerStarted","Data":"29ca861022bf520c2573e321b72dcc38755daa1eaebe8174caf60200234754e3"} Sep 17 01:41:50.990494 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:50.990462 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d848974c4-prkmc"] Sep 17 01:41:50.990699 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:41:50.990669 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5d848974c4-prkmc" podUID="d7cb31a9-2ccc-457e-9b48-60e7d5d92371" Sep 17 01:41:51.075131 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.075101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:51.175811 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.175659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:51.559915 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.559886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:51.560330 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.559916 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vkjcl" event={"ID":"bf91a16c-6494-4e30-a6e2-1cdefa9856c6","Type":"ContainerStarted","Data":"27f1663e44dfc681e2b3f151e93b23eafd550b5dc7aad2b61beb0b67312c47c8"} Sep 17 01:41:51.560330 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.559994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vkjcl" event={"ID":"bf91a16c-6494-4e30-a6e2-1cdefa9856c6","Type":"ContainerStarted","Data":"a7ce05cbac81acadda11ad9f7173a1d91a0d6245b4f8e4e569ddd4518d70ad63"} Sep 17 01:41:51.566620 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.566597 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:51.573675 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.573629 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vkjcl" podStartSLOduration=254.468478575 podStartE2EDuration="4m15.573607736s" totalImagesPullingTime="1.105129161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:51.573283337 +0000 UTC m=+256.408335671" watchObservedRunningTime="2026-09-17 01:41:51.573607736 +0000 UTC m=+256.408660068" Sep 17 01:41:51.647947 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.647922 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t45rr\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-kube-api-access-t45rr\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.647963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-installation-pull-secrets\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.647995 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-trusted-ca\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648031 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-certificates\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648234 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648057 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-bound-sa-token\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648234 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648083 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-image-registry-private-configuration\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648234 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648128 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-ca-trust-extracted\") pod \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\" (UID: \"d7cb31a9-2ccc-457e-9b48-60e7d5d92371\") " Sep 17 01:41:51.648596 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:51.648596 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648566 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:41:51.648920 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.648709 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:51.650494 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.650457 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:41:51.650494 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.650472 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:41:51.650641 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.650625 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-kube-api-access-t45rr" (OuterVolumeSpecName: "kube-api-access-t45rr") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "kube-api-access-t45rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:51.650677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.650658 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d7cb31a9-2ccc-457e-9b48-60e7d5d92371" (UID: "d7cb31a9-2ccc-457e-9b48-60e7d5d92371"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:51.655018 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.654970 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:51.748708 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748681 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45rr\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-kube-api-access-t45rr\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.748708 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748704 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-installation-pull-secrets\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.748837 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748718 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-trusted-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.748837 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748731 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-certificates\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.748837 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748743 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-bound-sa-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.748837 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748755 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-image-registry-private-configuration\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.748837 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.748767 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-ca-trust-extracted\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:51.755594 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.755562 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:51.856502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:51.856452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:52.562817 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:52.562785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d848974c4-prkmc" Sep 17 01:41:52.591688 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:52.591664 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d848974c4-prkmc"] Sep 17 01:41:52.595367 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:52.595341 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d848974c4-prkmc"] Sep 17 01:41:52.655290 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:52.655269 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7cb31a9-2ccc-457e-9b48-60e7d5d92371-registry-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:41:53.676753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:53.676721 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7cb31a9-2ccc-457e-9b48-60e7d5d92371" path="/var/lib/kubelet/pods/d7cb31a9-2ccc-457e-9b48-60e7d5d92371/volumes" Sep 17 01:41:53.929573 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:53.929500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:53.981253 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:53.981223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:57.213118 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.213080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:57.310609 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.310577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:57.368745 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.368714 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq"] Sep 17 01:41:57.374559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.374539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:57.376796 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.376778 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 01:41:57.376876 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.376829 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-972jl\"" Sep 17 01:41:57.379141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.379112 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq"] Sep 17 01:41:57.382806 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.382775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:57.489850 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.489802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8625ef2c-2727-4c73-94d5-de6ef4000ec8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-5jtkq\" (UID: \"8625ef2c-2727-4c73-94d5-de6ef4000ec8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:57.542614 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.542581 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:41:57.590568 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.590547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8625ef2c-2727-4c73-94d5-de6ef4000ec8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-5jtkq\" (UID: \"8625ef2c-2727-4c73-94d5-de6ef4000ec8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:57.592922 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.592893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8625ef2c-2727-4c73-94d5-de6ef4000ec8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-5jtkq\" (UID: \"8625ef2c-2727-4c73-94d5-de6ef4000ec8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:57.684270 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.684252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:57.948770 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:57.948595 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq"] Sep 17 01:41:57.951232 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:41:57.951195 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8625ef2c_2727_4c73_94d5_de6ef4000ec8.slice/crio-85885dbdcdd549d57d8c7843150aa3a628273fc36ec155a8aec04e9e3097916a WatchSource:0}: Error finding container 85885dbdcdd549d57d8c7843150aa3a628273fc36ec155a8aec04e9e3097916a: Status 404 returned error can't find the container with id 85885dbdcdd549d57d8c7843150aa3a628273fc36ec155a8aec04e9e3097916a Sep 17 01:41:58.586006 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:58.585975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" event={"ID":"8625ef2c-2727-4c73-94d5-de6ef4000ec8","Type":"ContainerStarted","Data":"85885dbdcdd549d57d8c7843150aa3a628273fc36ec155a8aec04e9e3097916a"} Sep 17 01:41:59.403184 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.403100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:59.405390 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.405357 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn297\" (UID: \"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc\") " pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:59.407872 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.407849 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4q7v\"" Sep 17 01:41:59.415804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.415782 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hn297" Sep 17 01:41:59.591267 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.591240 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" event={"ID":"8625ef2c-2727-4c73-94d5-de6ef4000ec8","Type":"ContainerStarted","Data":"0e2b0a41d9f877ced58e1c47b73218d9b65fb7c28af80a7fd37a359848755370"} Sep 17 01:41:59.591619 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.591413 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:59.596233 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.596202 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" Sep 17 01:41:59.605891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.605853 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-5jtkq" podStartSLOduration=1.561158688 podStartE2EDuration="2.605840137s" totalImagesPullingTime="1.044681449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:59.604904711 +0000 UTC m=+264.439957043" watchObservedRunningTime="2026-09-17 01:41:59.605840137 +0000 UTC m=+264.440892469" Sep 17 01:41:59.685323 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:41:59.685262 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hn297"] Sep 17 01:41:59.687867 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:41:59.687842 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf108a8c3_f01c_4c0a_ae6e_c9336ce4ffbc.slice/crio-2ed727ee6378aa492ccc6cbca80111601a7db1693c4d02c815b18731de8cd31c WatchSource:0}: Error finding container 2ed727ee6378aa492ccc6cbca80111601a7db1693c4d02c815b18731de8cd31c: Status 404 returned error can't find the container with id 2ed727ee6378aa492ccc6cbca80111601a7db1693c4d02c815b18731de8cd31c Sep 17 01:42:00.398263 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.398205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:00.421642 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.421606 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8"] Sep 17 01:42:00.424531 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.424512 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.426943 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.426923 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 01:42:00.426943 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.426939 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 17 01:42:00.427113 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.426941 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 17 01:42:00.428022 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.428004 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5nbff\"" Sep 17 01:42:00.428077 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.428029 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 01:42:00.428365 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.428350 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 01:42:00.432805 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.432779 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8"] Sep 17 01:42:00.497796 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.497755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:00.510176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.510156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.510273 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.510197 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65wd\" (UniqueName: \"kubernetes.io/projected/2d4f62d6-50d2-4dec-a191-da64d94c1362-kube-api-access-v65wd\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.510273 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.510263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4f62d6-50d2-4dec-a191-da64d94c1362-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.510343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.510298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.597227 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.597183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn297" event={"ID":"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc","Type":"ContainerStarted","Data":"d3b5de9b6a59c25c3c22d02c85c44e33aecf03cc1cadf22c11ec06088b50a69c"} Sep 17 01:42:00.597574 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.597249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn297" event={"ID":"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc","Type":"ContainerStarted","Data":"3abfc4372aa1b9b9e94f3da52a2b36b3eb1e59cf5f138383b446dc9501da7437"} Sep 17 01:42:00.597574 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.597273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn297" event={"ID":"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc","Type":"ContainerStarted","Data":"2ed727ee6378aa492ccc6cbca80111601a7db1693c4d02c815b18731de8cd31c"} Sep 17 01:42:00.610759 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.610733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.610835 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.610785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.610835 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.610811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65wd\" (UniqueName: \"kubernetes.io/projected/2d4f62d6-50d2-4dec-a191-da64d94c1362-kube-api-access-v65wd\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.610951 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:42:00.610933 2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 17 01:42:00.610986 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.610974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4f62d6-50d2-4dec-a191-da64d94c1362-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.611021 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:42:00.611001 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-tls podName:2d4f62d6-50d2-4dec-a191-da64d94c1362 nodeName:}" failed. No retries permitted until 2026-09-17 01:42:01.110980008 +0000 UTC m=+265.946032339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-5vbf8" (UID: "2d4f62d6-50d2-4dec-a191-da64d94c1362") : secret "prometheus-operator-tls" not found Sep 17 01:42:00.611576 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.611554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4f62d6-50d2-4dec-a191-da64d94c1362-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.613154 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.613131 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.618786 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.618760 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65wd\" (UniqueName: \"kubernetes.io/projected/2d4f62d6-50d2-4dec-a191-da64d94c1362-kube-api-access-v65wd\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:00.662433 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:00.662373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:01.113814 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:01.113781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:01.116545 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:01.116512 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2d4f62d6-50d2-4dec-a191-da64d94c1362-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5vbf8\" (UID: \"2d4f62d6-50d2-4dec-a191-da64d94c1362\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:01.335794 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:01.335758 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" Sep 17 01:42:01.610041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:01.609947 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8"] Sep 17 01:42:01.865317 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:42:01.865288 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4f62d6_50d2_4dec_a191_da64d94c1362.slice/crio-625d22ff57e07829b9e71b831fa5971b990196ffae37a284d5289257c9064632 WatchSource:0}: Error finding container 625d22ff57e07829b9e71b831fa5971b990196ffae37a284d5289257c9064632: Status 404 returned error can't find the container with id 625d22ff57e07829b9e71b831fa5971b990196ffae37a284d5289257c9064632 Sep 17 01:42:02.608479 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:02.608434 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn297" event={"ID":"f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc","Type":"ContainerStarted","Data":"42173930de5c1831be4739949796435e4f024c5ee742ce21fa73f1430ae8f395"} Sep 17 01:42:02.610047 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:02.610007 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" event={"ID":"2d4f62d6-50d2-4dec-a191-da64d94c1362","Type":"ContainerStarted","Data":"625d22ff57e07829b9e71b831fa5971b990196ffae37a284d5289257c9064632"} Sep 17 01:42:02.625287 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:02.625238 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hn297" podStartSLOduration=33.51959696 podStartE2EDuration="35.625224483s" totalImagesPullingTime="2.105627523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:41:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:02.62435102 +0000 UTC m=+267.459403391" watchObservedRunningTime="2026-09-17 01:42:02.625224483 +0000 UTC m=+267.460276807" Sep 17 01:42:03.465952 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:03.465918 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:03.615552 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:03.615473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" event={"ID":"2d4f62d6-50d2-4dec-a191-da64d94c1362","Type":"ContainerStarted","Data":"f654a2f1911528fd28506672d2c7bc640ca6fced3b8b75c98339397ff1883575"} Sep 17 01:42:03.615552 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:03.615528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" event={"ID":"2d4f62d6-50d2-4dec-a191-da64d94c1362","Type":"ContainerStarted","Data":"4fd64cdbc0db568ec7f0c245abd7fdd25a863ea3cc6cc75f06395d3538e90d37"} Sep 17 01:42:03.630336 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:03.630291 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5vbf8" podStartSLOduration=2.272695798 podStartE2EDuration="3.630275165s" totalImagesPullingTime="1.357579367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:03.629488603 +0000 UTC m=+268.464540937" watchObservedRunningTime="2026-09-17 01:42:03.630275165 +0000 UTC m=+268.465327499" Sep 17 01:42:05.522422 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.522381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:05.622468 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.622437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:05.735342 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.735312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:05.770690 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.770663 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kxnxg"] Sep 17 01:42:05.774938 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.774891 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.777337 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.777315 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 01:42:05.777439 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.777349 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9bxmv\"" Sep 17 01:42:05.777819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.777803 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 01:42:05.777926 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.777809 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 01:42:05.834780 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.834741 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:05.846436 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-textfile\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846561 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846561 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-root\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846561 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-wtmp\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846721 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846568 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-sys\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846721 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-metrics-client-ca\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846721 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846636 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-tls\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846721 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.846721 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.846700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wq4l\" (UniqueName: \"kubernetes.io/projected/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-kube-api-access-9wq4l\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.898422 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.898400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:05.947358 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947334 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wq4l\" (UniqueName: \"kubernetes.io/projected/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-kube-api-access-9wq4l\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947449 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-textfile\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947449 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947534 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-root\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947534 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-wtmp\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947534 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-sys\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-metrics-client-ca\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-tls\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-sys\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-textfile\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-root\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947881 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.947834 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-wtmp\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.947945 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:42:05.947928 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 17 01:42:05.948010 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:42:05.947996 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-tls podName:fc7568ca-af7a-4d75-842b-dd358c7b1f3d nodeName:}" failed. No retries permitted until 2026-09-17 01:42:06.447976361 +0000 UTC m=+271.283028672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-tls") pod "node-exporter-kxnxg" (UID: "fc7568ca-af7a-4d75-842b-dd358c7b1f3d") : secret "node-exporter-tls" not found Sep 17 01:42:05.948163 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.948144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.948225 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.948197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-metrics-client-ca\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.949850 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.949833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.955440 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.954500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wq4l\" (UniqueName: \"kubernetes.io/projected/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-kube-api-access-9wq4l\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:05.999083 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:05.999046 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:06.098989 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.098904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:06.199540 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.199513 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:06.390598 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.390535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:06.450764 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.450736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-tls\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:06.453235 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.453195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc7568ca-af7a-4d75-842b-dd358c7b1f3d-node-exporter-tls\") pod \"node-exporter-kxnxg\" (UID: \"fc7568ca-af7a-4d75-842b-dd358c7b1f3d\") " pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:06.685605 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.685554 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxnxg" Sep 17 01:42:06.697980 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:42:06.697952 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7568ca_af7a_4d75_842b_dd358c7b1f3d.slice/crio-19ff19a8dc70dd4d644cd5b05c988f2b7cd9ff97efa3b5dfc00c015e7123cf0d WatchSource:0}: Error finding container 19ff19a8dc70dd4d644cd5b05c988f2b7cd9ff97efa3b5dfc00c015e7123cf0d: Status 404 returned error can't find the container with id 19ff19a8dc70dd4d644cd5b05c988f2b7cd9ff97efa3b5dfc00c015e7123cf0d Sep 17 01:42:06.749176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.749146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:06.883982 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:06.883954 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:07.187919 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:07.187867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:07.629873 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:07.629846 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxnxg" event={"ID":"fc7568ca-af7a-4d75-842b-dd358c7b1f3d","Type":"ContainerStarted","Data":"312bbf0b94e96120574e5f78aa912212835747e8ea6044588cb05d877f85ce98"} Sep 17 01:42:07.629991 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:07.629904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxnxg" event={"ID":"fc7568ca-af7a-4d75-842b-dd358c7b1f3d","Type":"ContainerStarted","Data":"19ff19a8dc70dd4d644cd5b05c988f2b7cd9ff97efa3b5dfc00c015e7123cf0d"} Sep 17 01:42:08.636352 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.636305 2582 generic.go:356] "Generic (PLEG): container finished" podID="fc7568ca-af7a-4d75-842b-dd358c7b1f3d" containerID="312bbf0b94e96120574e5f78aa912212835747e8ea6044588cb05d877f85ce98" exitCode=0 Sep 17 01:42:08.636844 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.636385 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxnxg" event={"ID":"fc7568ca-af7a-4d75-842b-dd358c7b1f3d","Type":"ContainerDied","Data":"312bbf0b94e96120574e5f78aa912212835747e8ea6044588cb05d877f85ce98"} Sep 17 01:42:08.663047 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.663011 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:08.727075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.727033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:08.750038 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.750006 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77769479f9-4b4bk"] Sep 17 01:42:08.754933 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.754905 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.757621 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.757593 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 01:42:08.757896 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.757872 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 01:42:08.757993 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.757922 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m4kcl\"" Sep 17 01:42:08.758160 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.758125 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 01:42:08.758246 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.758195 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5kurbcsn80lna\"" Sep 17 01:42:08.758407 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.758387 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 01:42:08.758867 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.758843 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 01:42:08.765757 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.765723 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77769479f9-4b4bk"] Sep 17 01:42:08.801725 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.801684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:08.870601 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870780 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-tls\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870780 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870780 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfd268f3-cac2-4cdd-add7-17120729f3fb-metrics-client-ca\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870780 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txl4m\" (UniqueName: \"kubernetes.io/projected/cfd268f3-cac2-4cdd-add7-17120729f3fb-kube-api-access-txl4m\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870780 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870773 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-grpc-tls\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.870990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.870837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.971989 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.971959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-tls\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfd268f3-cac2-4cdd-add7-17120729f3fb-metrics-client-ca\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txl4m\" (UniqueName: \"kubernetes.io/projected/cfd268f3-cac2-4cdd-add7-17120729f3fb-kube-api-access-txl4m\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972414 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-grpc-tls\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972414 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972319 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.972414 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.973006 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.972952 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfd268f3-cac2-4cdd-add7-17120729f3fb-metrics-client-ca\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.975235 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.974861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.975235 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.975178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-tls\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.975235 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.975185 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.975235 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.975197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.975489 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.975269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.975770 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.975752 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cfd268f3-cac2-4cdd-add7-17120729f3fb-secret-grpc-tls\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:08.979201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:08.979182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txl4m\" (UniqueName: \"kubernetes.io/projected/cfd268f3-cac2-4cdd-add7-17120729f3fb-kube-api-access-txl4m\") pod \"thanos-querier-77769479f9-4b4bk\" (UID: \"cfd268f3-cac2-4cdd-add7-17120729f3fb\") " pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:09.038008 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.037971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:09.066652 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.066609 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:09.340690 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.340581 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77769479f9-4b4bk"] Sep 17 01:42:09.343105 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:42:09.343076 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd268f3_cac2_4cdd_add7_17120729f3fb.slice/crio-933ebf5bf3c644dc8a50eb781ca6c51488ab02ce0d7448467353f0dc9b27af2e WatchSource:0}: Error finding container 933ebf5bf3c644dc8a50eb781ca6c51488ab02ce0d7448467353f0dc9b27af2e: Status 404 returned error can't find the container with id 933ebf5bf3c644dc8a50eb781ca6c51488ab02ce0d7448467353f0dc9b27af2e Sep 17 01:42:09.646114 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.646024 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxnxg" event={"ID":"fc7568ca-af7a-4d75-842b-dd358c7b1f3d","Type":"ContainerStarted","Data":"6ce1ede1652f8856cfec7cfd4d2ba43045ac992c842b2128f15bc11a80113502"} Sep 17 01:42:09.646609 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.646130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxnxg" event={"ID":"fc7568ca-af7a-4d75-842b-dd358c7b1f3d","Type":"ContainerStarted","Data":"223842725d9bcbc01a9152328bb00b7c509d7c3a7e9c632b20a2c449b18e8790"} Sep 17 01:42:09.647285 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.647263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"933ebf5bf3c644dc8a50eb781ca6c51488ab02ce0d7448467353f0dc9b27af2e"} Sep 17 01:42:09.663541 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:09.663465 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kxnxg" podStartSLOduration=3.773280551 podStartE2EDuration="4.663448027s" totalImagesPullingTime="825.924437ms" totalInitContainerRuntime="64.243039ms" isStatefulPod=true podCreationTimestamp="2026-09-17 01:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:09.663232078 +0000 UTC m=+274.498284412" watchObservedRunningTime="2026-09-17 01:42:09.663448027 +0000 UTC m=+274.498500361" Sep 17 01:42:10.530695 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:10.530641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:10.900844 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:10.900748 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:11.657513 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.657476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"b964bd30c45d749cbd8a55ac44139c1bf155b610c3f22560699e7807dceaeae4"} Sep 17 01:42:11.657691 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.657543 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"a33fd5d33e7dfd1ebeda2d408891e8b006b3cfcaae58a400d109fee7a48383dd"} Sep 17 01:42:11.657691 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.657569 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"8c955c8ced4a082c6a5b0b49c49a4e6e94a99dbe7fb68c5d905541abf70d62e9"} Sep 17 01:42:11.895695 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.895658 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:42:11.898238 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.897313 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:11.900666 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.900646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.904413 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.903092 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 01:42:11.904413 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.903346 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 01:42:11.904413 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.903651 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zb4nw\"" Sep 17 01:42:11.905000 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.904982 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 01:42:11.905113 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.905094 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 01:42:11.905344 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.905325 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 01:42:11.905344 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.905333 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 01:42:11.905517 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.904982 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 01:42:11.905517 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.905457 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 01:42:11.905977 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.905808 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 01:42:11.906055 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.906034 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 01:42:11.906267 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.906248 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 01:42:11.906350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.906318 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-50vrniotvncej\"" Sep 17 01:42:11.906445 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.906428 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 01:42:11.906500 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.906453 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 01:42:11.906666 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.906638 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 01:42:11.908074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.908022 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 01:42:11.915560 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.915513 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-config\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997943 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-web-config\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997974 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.997996 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-config-out\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.998272 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.999472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.999472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.999472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.999472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:11.999472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:11.998439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrbc\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-kube-api-access-flrbc\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.099475 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.099433 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.099658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.099481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-config\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.099658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.099506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-web-config\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.099658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.099535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.099658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.099557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.099658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.099589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100199 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100199 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100172 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-config-out\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100449 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100473 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrbc\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-kube-api-access-flrbc\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.100692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.105418 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.105393 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-web-config\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.106118 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.105901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.106264 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.106138 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-config\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.106681 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.106659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.106743 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.106661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.107257 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.107065 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.108716 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.108665 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.111248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.100375 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.111248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.101334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.111248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.109841 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.111248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.110878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrbc\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-kube-api-access-flrbc\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.111510 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.111376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.114691 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.112303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.115746 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.115694 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.116663 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.116635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.118386 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.118362 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.118482 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.118469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-config-out\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.118876 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.118856 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.214896 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.214874 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:12.507030 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.507008 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:42:12.508751 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:42:12.508727 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbf5546_1794_4fe7_8943_1bf051d59208.slice/crio-79a2dee90d8c8fc078cee766522475e986629bbcc69af8c03d57d66f1e3d2c06 WatchSource:0}: Error finding container 79a2dee90d8c8fc078cee766522475e986629bbcc69af8c03d57d66f1e3d2c06: Status 404 returned error can't find the container with id 79a2dee90d8c8fc078cee766522475e986629bbcc69af8c03d57d66f1e3d2c06 Sep 17 01:42:12.663697 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.663668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"8d0dc2da475b103f4d96a5abe518f0550cab65f1cf70116bca8c80b1efaba648"} Sep 17 01:42:12.663832 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.663741 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"4d77b869c549bbe422531779628e160a2453d4874b1ab701e56f901d593220bf"} Sep 17 01:42:12.663832 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.663770 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" event={"ID":"cfd268f3-cac2-4cdd-add7-17120729f3fb","Type":"ContainerStarted","Data":"a94daebf2ac582f6f2b060cfe17f49260dc8fbeb3c830261b3cdc2abc5915c4c"} Sep 17 01:42:12.663832 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.663815 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:12.664875 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:12.664857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"79a2dee90d8c8fc078cee766522475e986629bbcc69af8c03d57d66f1e3d2c06"} Sep 17 01:42:13.670060 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:13.669980 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" exitCode=0 Sep 17 01:42:13.670060 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:13.670035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} Sep 17 01:42:13.696993 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:13.696939 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" podStartSLOduration=3.033006788 podStartE2EDuration="5.696917919s" totalImagesPullingTime="2.663911131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:12.704729213 +0000 UTC m=+277.539781545" watchObservedRunningTime="2026-09-17 01:42:13.696917919 +0000 UTC m=+278.531970299" Sep 17 01:42:17.690550 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.690515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} Sep 17 01:42:17.691056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.690590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} Sep 17 01:42:17.691056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.690630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} Sep 17 01:42:17.691056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.690671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} Sep 17 01:42:17.691056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.690715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} Sep 17 01:42:17.691056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.690753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerStarted","Data":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} Sep 17 01:42:17.729021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.728965 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.707602255 podStartE2EDuration="6.728945656s" totalImagesPullingTime="4.005694724s" totalInitContainerRuntime="15.648677ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:17.726757021 +0000 UTC m=+282.561809395" watchObservedRunningTime="2026-09-17 01:42:17.728945656 +0000 UTC m=+282.563997991" Sep 17 01:42:17.734096 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.734069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:17.796239 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.796201 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:17.899275 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:17.899246 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:18.018476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.018446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:18.274022 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:42:18.273953 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q2l8r" podUID="cdcd7a2a-0e54-4fef-abb6-a1046f64fa21" Sep 17 01:42:18.390853 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.390824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:18.492266 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.492238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:18.676262 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.676181 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77769479f9-4b4bk" Sep 17 01:42:18.696344 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.696320 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q2l8r" Sep 17 01:42:18.790190 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.790144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:18.850594 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:18.850564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.021960 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.021926 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.123461 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.123431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.226367 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.226331 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.323652 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.323573 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.388444 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.388416 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.460993 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.460965 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.561362 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.561319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:19.661586 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:19.661517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:21.018434 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.018402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:21.118860 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.118829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:21.483559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.483532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:42:21.486062 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.486040 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cdcd7a2a-0e54-4fef-abb6-a1046f64fa21-metrics-tls\") pod \"dns-default-q2l8r\" (UID: \"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21\") " pod="openshift-dns/dns-default-q2l8r" Sep 17 01:42:21.699532 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.699507 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crsck\"" Sep 17 01:42:21.707522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.707503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q2l8r" Sep 17 01:42:21.982669 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:21.982642 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q2l8r"] Sep 17 01:42:21.984781 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:42:21.984753 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcd7a2a_0e54_4fef_abb6_a1046f64fa21.slice/crio-a03e425446dc819b7df746d2c98e4a7d464a11564335f14fe97f5dd91fa61851 WatchSource:0}: Error finding container a03e425446dc819b7df746d2c98e4a7d464a11564335f14fe97f5dd91fa61851: Status 404 returned error can't find the container with id a03e425446dc819b7df746d2c98e4a7d464a11564335f14fe97f5dd91fa61851 Sep 17 01:42:22.215281 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:22.215261 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:42:22.710800 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:22.710723 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q2l8r" event={"ID":"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21","Type":"ContainerStarted","Data":"a03e425446dc819b7df746d2c98e4a7d464a11564335f14fe97f5dd91fa61851"} Sep 17 01:42:23.412443 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:23.412380 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:23.466445 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:23.466410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:23.544745 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:23.544710 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:23.645228 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:23.645181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:24.088142 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:24.088091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:24.136325 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:24.136297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:24.719631 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:24.719598 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q2l8r" event={"ID":"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21","Type":"ContainerStarted","Data":"a974ff92d494705761bf2b9965a82bbe149264ff96145eb29d15cb394fa0b6fb"} Sep 17 01:42:24.720083 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:24.719653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q2l8r" event={"ID":"cdcd7a2a-0e54-4fef-abb6-a1046f64fa21","Type":"ContainerStarted","Data":"84327005ed42a9c083973d63c0d50aff96b6b8a915b6a50b491fa651f9a73dc9"} Sep 17 01:42:24.720083 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:24.719718 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q2l8r" Sep 17 01:42:24.736902 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:24.736862 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q2l8r" podStartSLOduration=252.065649609 podStartE2EDuration="4m13.73685121s" totalImagesPullingTime="1.671201601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:24.734684665 +0000 UTC m=+289.569736998" watchObservedRunningTime="2026-09-17 01:42:24.73685121 +0000 UTC m=+289.571903543" Sep 17 01:42:26.105862 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:26.105830 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:26.160173 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:26.160146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:26.851688 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:26.851653 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:26.903845 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:26.903800 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:33.167099 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:33.167001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:33.267732 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:33.267698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:34.620045 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:34.620008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:34.724927 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:34.724893 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q2l8r" Sep 17 01:42:34.725664 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:34.725630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:34.839969 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:34.839928 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:34.937485 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:34.937395 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:35.555070 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:35.555043 2582 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 01:42:38.618170 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:38.618139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:38.674632 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:38.674602 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:38.932811 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:38.932715 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:38.980452 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:38.980427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:44.225608 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:44.225569 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:44.283994 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:44.283968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:44.864594 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:44.864557 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:44.965035 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:44.964988 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:45.798499 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:45.798471 2582 generic.go:356] "Generic (PLEG): container finished" podID="5ad2fb6c-1185-4211-8e82-ae810c248e8b" containerID="a0fc23f99e20b2057f86d533526d7a521e8fbe8c0f90f32430d19da85fcdfa63" exitCode=0 Sep 17 01:42:45.798857 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:45.798548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" event={"ID":"5ad2fb6c-1185-4211-8e82-ae810c248e8b","Type":"ContainerDied","Data":"a0fc23f99e20b2057f86d533526d7a521e8fbe8c0f90f32430d19da85fcdfa63"} Sep 17 01:42:45.798857 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:45.798851 2582 scope.go:117] "RemoveContainer" containerID="a0fc23f99e20b2057f86d533526d7a521e8fbe8c0f90f32430d19da85fcdfa63" Sep 17 01:42:46.426801 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:46.426764 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:46.527369 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:46.527329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:46.803993 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:46.803961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7hm4g" event={"ID":"5ad2fb6c-1185-4211-8e82-ae810c248e8b","Type":"ContainerStarted","Data":"5abc2e192b8f7e0b6cb051e1178b6ab17602f3b9d408ec57ee200351e3c40f91"} Sep 17 01:42:48.657701 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:48.657654 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:48.759001 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:48.758968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:49.049794 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:49.049764 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:49.150965 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:49.150931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:49.700703 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:49.700671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:49.756148 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:49.756117 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:53.931519 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:53.931481 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:42:53.982884 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:42:53.982850 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:08.948580 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:08.948541 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:09.008039 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:09.008003 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:12.215278 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:12.215240 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:12.235742 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:12.235720 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:12.922224 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:12.922179 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:24.880599 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:24.880554 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:24.981528 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:24.981496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:30.227311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.227226 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:43:30.228230 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.228181 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="prometheus" containerID="cri-o://19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" gracePeriod=600 Sep 17 01:43:30.228619 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.228416 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="thanos-sidecar" containerID="cri-o://efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" gracePeriod=600 Sep 17 01:43:30.228721 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.228439 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-web" containerID="cri-o://b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" gracePeriod=600 Sep 17 01:43:30.228785 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.228466 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="config-reloader" containerID="cri-o://4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" gracePeriod=600 Sep 17 01:43:30.228841 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.228466 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy" containerID="cri-o://4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" gracePeriod=600 Sep 17 01:43:30.228897 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.228486 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" gracePeriod=600 Sep 17 01:43:30.479415 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.479356 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:30.600591 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600561 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flrbc\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-kube-api-access-flrbc\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600600 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-1\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600624 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-serving-certs-ca-bundle\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600668 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-kubelet-serving-ca-bundle\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600696 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600723 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-trusted-ca-bundle\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600751 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-tls\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600777 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-web-config\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-grpc-tls\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600839 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-config-out\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600881 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600914 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-tls-assets\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600918 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600947 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-0\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.600997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.600980 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-2\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-db\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601033 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-config\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601066 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601077 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-metrics-client-ca\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601097 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601128 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-thanos-prometheus-http-client-file\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601158 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-metrics-client-certs\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601196 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-kube-rbac-proxy\") pod \"6fbf5546-1794-4fe7-8943-1bf051d59208\" (UID: \"6fbf5546-1794-4fe7-8943-1bf051d59208\") " Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601390 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.601481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601442 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.601988 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601492 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.601988 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601511 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-metrics-client-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.601988 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601526 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.601988 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601541 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.601988 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.601794 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.602576 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.602550 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:43:30.602890 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.602856 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:43:30.603181 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.603150 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.603798 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.603770 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-kube-api-access-flrbc" (OuterVolumeSpecName: "kube-api-access-flrbc") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "kube-api-access-flrbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:43:30.603898 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.603879 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.604194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.604162 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.604302 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.604251 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.604757 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.604731 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.605262 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.605236 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-config" (OuterVolumeSpecName: "config") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.605349 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.605265 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.605683 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.605657 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-config-out" (OuterVolumeSpecName: "config-out") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:43:30.605877 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.605862 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.605975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.605961 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:43:30.615972 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.615950 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-web-config" (OuterVolumeSpecName: "web-config") pod "6fbf5546-1794-4fe7-8943-1bf051d59208" (UID: "6fbf5546-1794-4fe7-8943-1bf051d59208"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:43:30.702495 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702461 2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-grpc-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702495 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702492 2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-config-out\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702495 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702502 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702510 2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-tls-assets\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702521 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702530 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702538 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6fbf5546-1794-4fe7-8943-1bf051d59208-prometheus-k8s-db\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702546 2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702554 2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702564 2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-metrics-client-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702573 2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-kube-rbac-proxy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702582 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrbc\" (UniqueName: \"kubernetes.io/projected/6fbf5546-1794-4fe7-8943-1bf051d59208-kube-api-access-flrbc\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702590 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fbf5546-1794-4fe7-8943-1bf051d59208-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702599 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702608 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.702654 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.702617 2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6fbf5546-1794-4fe7-8943-1bf051d59208-web-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:43:30.978878 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978848 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" exitCode=0 Sep 17 01:43:30.978878 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978870 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" exitCode=0 Sep 17 01:43:30.978878 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978879 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" exitCode=0 Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978885 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" exitCode=0 Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978892 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" exitCode=0 Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978897 2582 generic.go:356] "Generic (PLEG): container finished" podID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" exitCode=0 Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978929 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.978931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979002 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} Sep 17 01:43:30.979046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} Sep 17 01:43:30.979311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979057 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} Sep 17 01:43:30.979311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979070 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:30.979311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} Sep 17 01:43:30.979311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} Sep 17 01:43:30.979311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.979133 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6fbf5546-1794-4fe7-8943-1bf051d59208","Type":"ContainerDied","Data":"79a2dee90d8c8fc078cee766522475e986629bbcc69af8c03d57d66f1e3d2c06"} Sep 17 01:43:30.987473 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.987459 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:30.995143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:30.995125 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.002408 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.002385 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.003122 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.003093 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:43:31.009733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.009605 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:43:31.009798 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.009753 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.016340 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.016322 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.023098 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.023084 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.029522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.029507 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.029792 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.029773 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.029854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.029800 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} err="failed to get container status \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" Sep 17 01:43:31.029854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.029828 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.030061 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.030039 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.030105 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030068 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} err="failed to get container status \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" Sep 17 01:43:31.030105 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030088 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.030369 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.030349 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.030429 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030373 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} err="failed to get container status \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" Sep 17 01:43:31.030429 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030388 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.030646 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.030632 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.030687 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030650 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} err="failed to get container status \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" Sep 17 01:43:31.030687 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030672 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.030915 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.030897 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.030951 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030919 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} err="failed to get container status \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" Sep 17 01:43:31.030951 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.030934 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.031144 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.031127 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.031185 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031147 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} err="failed to get container status \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" Sep 17 01:43:31.031185 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031158 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.031393 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:43:31.031378 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.031435 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031396 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} err="failed to get container status \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" Sep 17 01:43:31.031435 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031416 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.031656 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031633 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} err="failed to get container status \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" Sep 17 01:43:31.031711 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031655 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.031884 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031866 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} err="failed to get container status \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" Sep 17 01:43:31.031926 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.031885 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.032150 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032134 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} err="failed to get container status \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" Sep 17 01:43:31.032204 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032150 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.032388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032372 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} err="failed to get container status \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" Sep 17 01:43:31.032388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032388 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.032595 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032577 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} err="failed to get container status \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" Sep 17 01:43:31.032637 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032596 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.032859 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032802 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} err="failed to get container status \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" Sep 17 01:43:31.032937 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.032861 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.033114 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.033089 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} err="failed to get container status \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" Sep 17 01:43:31.033285 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.033115 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.033458 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.033432 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} err="failed to get container status \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" Sep 17 01:43:31.033523 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.033459 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.033773 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.033749 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} err="failed to get container status \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" Sep 17 01:43:31.033773 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.033772 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.034546 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.034520 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} err="failed to get container status \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" Sep 17 01:43:31.034546 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.034547 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.034896 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.034873 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} err="failed to get container status \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" Sep 17 01:43:31.034991 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.034897 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.035040 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.034986 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035345 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="prometheus" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035365 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="prometheus" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035390 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="config-reloader" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035398 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="config-reloader" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035412 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-web" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035421 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-web" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035430 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035438 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035450 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="init-config-reloader" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035458 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="init-config-reloader" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035471 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-thanos" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035478 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-thanos" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035501 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="thanos-sidecar" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035573 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="thanos-sidecar" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035655 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-thanos" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035666 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="config-reloader" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035676 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="prometheus" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035685 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="thanos-sidecar" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035694 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy" Sep 17 01:43:31.035729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.035705 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" containerName="kube-rbac-proxy-web" Sep 17 01:43:31.038227 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.038190 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} err="failed to get container status \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" Sep 17 01:43:31.038414 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.038232 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.038862 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.038802 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} err="failed to get container status \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" Sep 17 01:43:31.038862 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.038831 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.039196 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.039162 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} err="failed to get container status \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" Sep 17 01:43:31.039196 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.039196 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.039593 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.039573 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} err="failed to get container status \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" Sep 17 01:43:31.039682 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.039594 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.040335 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.039885 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} err="failed to get container status \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" Sep 17 01:43:31.040335 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.039912 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.041666 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.041591 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.042176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042001 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} err="failed to get container status \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" Sep 17 01:43:31.042176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042029 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.042408 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042387 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} err="failed to get container status \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" Sep 17 01:43:31.042458 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042414 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.042672 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042656 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} err="failed to get container status \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" Sep 17 01:43:31.042733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042673 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.042926 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042910 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} err="failed to get container status \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" Sep 17 01:43:31.042985 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.042926 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.043173 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.043154 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} err="failed to get container status \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" Sep 17 01:43:31.043252 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.043174 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.043515 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.043479 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} err="failed to get container status \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" Sep 17 01:43:31.043596 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.043516 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.043840 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.043809 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} err="failed to get container status \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" Sep 17 01:43:31.043975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.043843 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.044153 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044119 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 01:43:31.044284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044170 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 01:43:31.044284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044192 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-50vrniotvncej\"" Sep 17 01:43:31.044443 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044425 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 01:43:31.044548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044428 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 01:43:31.044548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044460 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 01:43:31.044548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044475 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 01:43:31.044548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044483 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 01:43:31.044548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044483 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zb4nw\"" Sep 17 01:43:31.044851 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044554 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 01:43:31.044851 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044673 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} err="failed to get container status \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" Sep 17 01:43:31.044851 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044697 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.045074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044914 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 01:43:31.045074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.044980 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} err="failed to get container status \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" Sep 17 01:43:31.045074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045002 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.045380 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045294 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} err="failed to get container status \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" Sep 17 01:43:31.045380 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045316 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.045538 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045492 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 01:43:31.045643 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045620 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} err="failed to get container status \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" Sep 17 01:43:31.045854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045647 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.045854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045498 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 01:43:31.046055 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045922 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 01:43:31.046055 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045936 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} err="failed to get container status \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" Sep 17 01:43:31.046055 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.045962 2582 scope.go:117] "RemoveContainer" containerID="0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5" Sep 17 01:43:31.047149 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.046773 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 01:43:31.051687 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.050456 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 01:43:31.051964 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.051934 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5"} err="failed to get container status \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": rpc error: code = NotFound desc = could not find container \"0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5\": container with ID starting with 0ec4f8027143c650a2aabdc092fbd7da150b4ac4248644acac4911dd134161e5 not found: ID does not exist" Sep 17 01:43:31.051964 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.051965 2582 scope.go:117] "RemoveContainer" containerID="4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25" Sep 17 01:43:31.052746 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.052717 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25"} err="failed to get container status \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": rpc error: code = NotFound desc = could not find container \"4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25\": container with ID starting with 4e48c121cba9c25b46b93130d5e7cef29574617bcd8b1ed15d03d50eab6d7b25 not found: ID does not exist" Sep 17 01:43:31.052746 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.052747 2582 scope.go:117] "RemoveContainer" containerID="b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711" Sep 17 01:43:31.052916 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.052896 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 01:43:31.053058 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.053028 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711"} err="failed to get container status \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": rpc error: code = NotFound desc = could not find container \"b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711\": container with ID starting with b5cfec5043d9cc1fff3bb68c1e852c34accaff0d31b9e4413971a1480026d711 not found: ID does not exist" Sep 17 01:43:31.053170 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.053060 2582 scope.go:117] "RemoveContainer" containerID="efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1" Sep 17 01:43:31.053489 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.053346 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1"} err="failed to get container status \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": rpc error: code = NotFound desc = could not find container \"efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1\": container with ID starting with efbef89fddf8a1bc849de9c014c45239342a7ebf50389c2973117375015230b1 not found: ID does not exist" Sep 17 01:43:31.053489 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.053389 2582 scope.go:117] "RemoveContainer" containerID="4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725" Sep 17 01:43:31.054490 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.054466 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725"} err="failed to get container status \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": rpc error: code = NotFound desc = could not find container \"4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725\": container with ID starting with 4be4f990b0a124b9d5d8f86d17b708a4b6a8271bc02a33f3cc6ac6ca01847725 not found: ID does not exist" Sep 17 01:43:31.054560 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.054490 2582 scope.go:117] "RemoveContainer" containerID="19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62" Sep 17 01:43:31.055148 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.055076 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62"} err="failed to get container status \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": rpc error: code = NotFound desc = could not find container \"19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62\": container with ID starting with 19f716eda44ce9c11e4bda7fd60f477f8a7db6adfe56358c3d083fc83dfaca62 not found: ID does not exist" Sep 17 01:43:31.055148 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.055129 2582 scope.go:117] "RemoveContainer" containerID="0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817" Sep 17 01:43:31.055646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.055612 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817"} err="failed to get container status \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": rpc error: code = NotFound desc = could not find container \"0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817\": container with ID starting with 0422d48d8f69e5f747c72f4fabc8fde5e951a4c04e6462de869f1a80fd5e4817 not found: ID does not exist" Sep 17 01:43:31.056226 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.056193 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:43:31.207036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-web-config\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-config\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxld\" (UniqueName: \"kubernetes.io/projected/71d285eb-613e-49bd-9559-d7fce5e94b03-kube-api-access-llxld\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71d285eb-613e-49bd-9559-d7fce5e94b03-config-out\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71d285eb-613e-49bd-9559-d7fce5e94b03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.207706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.207623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.307872 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.307847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71d285eb-613e-49bd-9559-d7fce5e94b03-config-out\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.307880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.307907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71d285eb-613e-49bd-9559-d7fce5e94b03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308240 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-web-config\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-config\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxld\" (UniqueName: \"kubernetes.io/projected/71d285eb-613e-49bd-9559-d7fce5e94b03-kube-api-access-llxld\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308434 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.308766 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.309271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.308814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.309811 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.309495 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311007 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.310684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71d285eb-613e-49bd-9559-d7fce5e94b03-config-out\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311007 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.310923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71d285eb-613e-49bd-9559-d7fce5e94b03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311159 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.311034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.311482 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.311557 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.311659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-config\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.311992 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.311835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.312102 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.312077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.312161 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.312118 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.312258 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.312227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.312450 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.312383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.312680 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.312632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/71d285eb-613e-49bd-9559-d7fce5e94b03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.313391 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.313361 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.313487 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.313402 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.313874 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.313857 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-web-config\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.314264 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.314242 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.314693 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.314676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71d285eb-613e-49bd-9559-d7fce5e94b03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.323462 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.323440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxld\" (UniqueName: \"kubernetes.io/projected/71d285eb-613e-49bd-9559-d7fce5e94b03-kube-api-access-llxld\") pod \"prometheus-k8s-0\" (UID: \"71d285eb-613e-49bd-9559-d7fce5e94b03\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.357342 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.357321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:31.636863 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.636836 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 01:43:31.638241 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:43:31.638202 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d285eb_613e_49bd_9559_d7fce5e94b03.slice/crio-02cbc176153e7da6f88c7407fdcba4de4e6d9302086849c4bae5cafee9592356 WatchSource:0}: Error finding container 02cbc176153e7da6f88c7407fdcba4de4e6d9302086849c4bae5cafee9592356: Status 404 returned error can't find the container with id 02cbc176153e7da6f88c7407fdcba4de4e6d9302086849c4bae5cafee9592356 Sep 17 01:43:31.678716 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.678692 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbf5546-1794-4fe7-8943-1bf051d59208" path="/var/lib/kubelet/pods/6fbf5546-1794-4fe7-8943-1bf051d59208/volumes" Sep 17 01:43:31.984281 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.984254 2582 generic.go:356] "Generic (PLEG): container finished" podID="71d285eb-613e-49bd-9559-d7fce5e94b03" containerID="5be6ca3c13d38cc454edac09f9991342b9548302abce04ac116684034e91fbce" exitCode=0 Sep 17 01:43:31.984402 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.984348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerDied","Data":"5be6ca3c13d38cc454edac09f9991342b9548302abce04ac116684034e91fbce"} Sep 17 01:43:31.984456 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:31.984421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"02cbc176153e7da6f88c7407fdcba4de4e6d9302086849c4bae5cafee9592356"} Sep 17 01:43:32.232636 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.232606 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:32.334175 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.334151 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:32.994074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.994034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"0e98898122e98900c410778fa5a33327a86f77516799d9359d87a6319d8949ee"} Sep 17 01:43:32.994247 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.994110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"5bb0451fe2a33f8b0e99e83f60ce6db3526f9568f63bc27e08b51540ea574054"} Sep 17 01:43:32.994247 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.994138 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"75d5e4394516dbba1989033d0a28f10f5b62276e81ba0cd9f4d9b68cd32a1239"} Sep 17 01:43:32.994247 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.994165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"e242dd8977469ee1fbc2509f7eb2b88f7799254dc94c10a2719e3b41e5409a77"} Sep 17 01:43:32.994247 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.994203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"cc0456e14738cb97c4d7141609eddac14495af7909339a59ab8f7cfff70f00e1"} Sep 17 01:43:32.994247 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:32.994249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"71d285eb-613e-49bd-9559-d7fce5e94b03","Type":"ContainerStarted","Data":"325e9d896c4504868efb82636faf63b904d60f32564854a58aee483566336d79"} Sep 17 01:43:33.028230 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:33.024633 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.00675579 podStartE2EDuration="2.024608849s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.853059ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:33.020505826 +0000 UTC m=+357.855558179" watchObservedRunningTime="2026-09-17 01:43:33.024608849 +0000 UTC m=+357.859661184" Sep 17 01:43:33.306040 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:33.306010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:33.361912 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:33.361880 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:36.357994 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:36.357960 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:43:38.791046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:38.791010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:38.841748 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:38.841714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:43.856333 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:43.856296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:43.957938 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:43.957909 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:43:44.009893 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:43:44.009856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:13.906232 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:13.906173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:13.960250 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:13.960222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:31.357506 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:31.357466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:44:31.374460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:31.374429 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:44:32.243518 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:32.243489 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 01:44:40.087262 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:40.087200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:40.143952 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:40.143921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:43.622801 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:43.622760 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:43.673610 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:43.673573 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:52.051198 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:52.051159 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:52.114116 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:52.114076 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:54.134965 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:54.134927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:54.234980 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:54.234947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:44:54.335458 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:44:54.335428 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:05.332308 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:05.332198 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:05.394101 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:05.394067 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:08.941168 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:08.941121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:09.043627 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:09.043579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:09.348384 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:09.348344 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:44.418855 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:44.418810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:50.035067 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:50.035033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:45:50.137052 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:45:50.137019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:11.670711 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:11.670672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:11.884485 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:11.884454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:12.696638 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:12.696598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:15.294027 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:15.293992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:15.394299 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:15.394255 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:16.250559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:16.250523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:16.380201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:16.380153 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:17.454503 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:17.454468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:17.555894 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:17.555863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:19.678155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:19.678109 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:19.868499 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:19.868467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:22.091968 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:22.091929 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:22.187395 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:22.187353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:24.445968 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:24.445933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:27.884660 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:27.884575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:28.102264 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:28.102231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:28.200402 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:28.200323 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:35.017791 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:35.017756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:35.121338 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:35.121303 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:36.845739 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:36.845703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:49.145671 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.145620 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:49.236319 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.236279 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl"] Sep 17 01:46:49.239742 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.239710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.243390 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.243358 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-djpgp\"" Sep 17 01:46:49.243586 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.243390 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 01:46:49.243586 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.243430 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 17 01:46:49.243586 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.243466 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 17 01:46:49.243586 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.243358 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:46:49.243586 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.243529 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 17 01:46:49.244644 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.244615 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:49.248388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.248349 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl"] Sep 17 01:46:49.349544 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.349496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/498ead04-90ec-45ba-bafb-aebafb2fda1c-manager-config\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.349544 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.349544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/498ead04-90ec-45ba-bafb-aebafb2fda1c-cert\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.349782 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.349569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/498ead04-90ec-45ba-bafb-aebafb2fda1c-metrics-cert\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.349782 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.349639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtk2\" (UniqueName: \"kubernetes.io/projected/498ead04-90ec-45ba-bafb-aebafb2fda1c-kube-api-access-hvtk2\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.424313 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.424225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:49.451058 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.451016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/498ead04-90ec-45ba-bafb-aebafb2fda1c-manager-config\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.451258 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.451065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/498ead04-90ec-45ba-bafb-aebafb2fda1c-cert\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.451258 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.451105 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/498ead04-90ec-45ba-bafb-aebafb2fda1c-metrics-cert\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.451258 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.451145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtk2\" (UniqueName: \"kubernetes.io/projected/498ead04-90ec-45ba-bafb-aebafb2fda1c-kube-api-access-hvtk2\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.451843 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.451815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/498ead04-90ec-45ba-bafb-aebafb2fda1c-manager-config\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.453910 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.453882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/498ead04-90ec-45ba-bafb-aebafb2fda1c-cert\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.454014 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.453882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/498ead04-90ec-45ba-bafb-aebafb2fda1c-metrics-cert\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.470155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.470118 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtk2\" (UniqueName: \"kubernetes.io/projected/498ead04-90ec-45ba-bafb-aebafb2fda1c-kube-api-access-hvtk2\") pod \"lws-controller-manager-7c846775dd-cdsbl\" (UID: \"498ead04-90ec-45ba-bafb-aebafb2fda1c\") " pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.529902 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.529860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:49.550868 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.550829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:49.827060 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.827033 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl"] Sep 17 01:46:49.829709 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:46:49.829673 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498ead04_90ec_45ba_bafb_aebafb2fda1c.slice/crio-0b1e2d914452025658cddfd87f38e655725a6f3047153e5ef3d6798429743264 WatchSource:0}: Error finding container 0b1e2d914452025658cddfd87f38e655725a6f3047153e5ef3d6798429743264: Status 404 returned error can't find the container with id 0b1e2d914452025658cddfd87f38e655725a6f3047153e5ef3d6798429743264 Sep 17 01:46:49.831643 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:49.831625 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:46:50.089343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.089245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:50.298577 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.298542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:50.393771 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.393671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:50.441371 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.441340 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr"] Sep 17 01:46:50.446132 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.446110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.450676 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.450647 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 01:46:50.451318 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.451302 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 01:46:50.451421 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.451402 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 01:46:50.451476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.451459 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 01:46:50.451667 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.451644 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-994j7\"" Sep 17 01:46:50.461874 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.461842 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr"] Sep 17 01:46:50.560863 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.560831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14d1372a-5a9e-4db3-8768-5ad476bf3476-tmp\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.561041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.560875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14d1372a-5a9e-4db3-8768-5ad476bf3476-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.561041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.561014 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14d1372a-5a9e-4db3-8768-5ad476bf3476-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.561157 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.561064 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttj22\" (UniqueName: \"kubernetes.io/projected/14d1372a-5a9e-4db3-8768-5ad476bf3476-kube-api-access-ttj22\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.561233 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.561162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14d1372a-5a9e-4db3-8768-5ad476bf3476-manifests\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.604640 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.604601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:50.661967 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.661878 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14d1372a-5a9e-4db3-8768-5ad476bf3476-tmp\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.661967 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.661927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14d1372a-5a9e-4db3-8768-5ad476bf3476-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.661967 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.661963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14d1372a-5a9e-4db3-8768-5ad476bf3476-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.662251 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.661992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttj22\" (UniqueName: \"kubernetes.io/projected/14d1372a-5a9e-4db3-8768-5ad476bf3476-kube-api-access-ttj22\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.662251 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.662043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14d1372a-5a9e-4db3-8768-5ad476bf3476-manifests\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.662524 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.662500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14d1372a-5a9e-4db3-8768-5ad476bf3476-manifests\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.664703 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.664677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14d1372a-5a9e-4db3-8768-5ad476bf3476-tmp\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.664817 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.664778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14d1372a-5a9e-4db3-8768-5ad476bf3476-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.664972 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.664949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14d1372a-5a9e-4db3-8768-5ad476bf3476-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.669991 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.669965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttj22\" (UniqueName: \"kubernetes.io/projected/14d1372a-5a9e-4db3-8768-5ad476bf3476-kube-api-access-ttj22\") pod \"opendatahub-operator-controller-manager-7f5f459ff6-d8jlr\" (UID: \"14d1372a-5a9e-4db3-8768-5ad476bf3476\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.705987 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.705921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:50.763950 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.763915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:50.774231 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.774197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" event={"ID":"498ead04-90ec-45ba-bafb-aebafb2fda1c","Type":"ContainerStarted","Data":"0b1e2d914452025658cddfd87f38e655725a6f3047153e5ef3d6798429743264"} Sep 17 01:46:50.806523 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.806460 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:50.908042 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:50.908009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:51.000490 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:51.000461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:46:51.053223 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:51.053180 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr"] Sep 17 01:46:51.057525 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:46:51.057488 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d1372a_5a9e_4db3_8768_5ad476bf3476.slice/crio-70e3da4742082e6e38a4115d3d61ad77e7037b58203250ce2949110c8030175f WatchSource:0}: Error finding container 70e3da4742082e6e38a4115d3d61ad77e7037b58203250ce2949110c8030175f: Status 404 returned error can't find the container with id 70e3da4742082e6e38a4115d3d61ad77e7037b58203250ce2949110c8030175f Sep 17 01:46:51.779785 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:51.779750 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" event={"ID":"14d1372a-5a9e-4db3-8768-5ad476bf3476","Type":"ContainerStarted","Data":"70e3da4742082e6e38a4115d3d61ad77e7037b58203250ce2949110c8030175f"} Sep 17 01:46:52.789449 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:52.789409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" event={"ID":"498ead04-90ec-45ba-bafb-aebafb2fda1c","Type":"ContainerStarted","Data":"72872f85bcd661b080c90b91bfd5bef4cdfa27f195bde4a0f3263cd3139932d3"} Sep 17 01:46:52.790176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:52.790138 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:46:52.812944 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:52.812712 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" podStartSLOduration=1.3478592950000001 podStartE2EDuration="3.812692626s" totalImagesPullingTime="2.464833331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:52.811175164 +0000 UTC m=+557.646227498" watchObservedRunningTime="2026-09-17 01:46:52.812692626 +0000 UTC m=+557.647744960" Sep 17 01:46:54.799257 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:54.799230 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" event={"ID":"14d1372a-5a9e-4db3-8768-5ad476bf3476","Type":"ContainerStarted","Data":"33be2939f118eb7e5aed0fac3680a3ce69c86177437a6e626a36592964cad82f"} Sep 17 01:46:55.804734 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:55.804702 2582 generic.go:356] "Generic (PLEG): container finished" podID="14d1372a-5a9e-4db3-8768-5ad476bf3476" containerID="33be2939f118eb7e5aed0fac3680a3ce69c86177437a6e626a36592964cad82f" exitCode=0 Sep 17 01:46:55.805075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:55.804784 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" event={"ID":"14d1372a-5a9e-4db3-8768-5ad476bf3476","Type":"ContainerDied","Data":"33be2939f118eb7e5aed0fac3680a3ce69c86177437a6e626a36592964cad82f"} Sep 17 01:46:56.810632 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:56.810601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" event={"ID":"14d1372a-5a9e-4db3-8768-5ad476bf3476","Type":"ContainerStarted","Data":"9b322a7835fc765479160e8cc67b5f6562a0a6f9f0af43cd3515f3dd0251a386"} Sep 17 01:46:56.810979 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:56.810719 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:46:56.831893 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:46:56.831852 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" podStartSLOduration=1.8784858500000001 podStartE2EDuration="6.831839221s" totalImagesPullingTime="3.965247555s" totalInitContainerRuntime="988.105816ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:46:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:46:56.830396161 +0000 UTC m=+561.665448497" watchObservedRunningTime="2026-09-17 01:46:56.831839221 +0000 UTC m=+561.666891553" Sep 17 01:47:01.079434 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:01.079399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:01.180853 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:01.180810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:04.800942 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:04.800913 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c846775dd-cdsbl" Sep 17 01:47:07.816946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:07.816917 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5f459ff6-d8jlr" Sep 17 01:47:09.640651 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:09.640612 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:09.764176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:09.764138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:10.014108 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.014074 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:10.150906 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.150870 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x"] Sep 17 01:47:10.161652 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.161619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.165575 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.165549 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 01:47:10.165723 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.165635 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 01:47:10.165781 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.165732 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 01:47:10.166511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.166495 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 01:47:10.166511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.166503 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zthpz\"" Sep 17 01:47:10.173957 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.173934 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x"] Sep 17 01:47:10.215892 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.215857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653dc08d-064b-469c-8583-b87821c2d482-tmp\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.215892 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.215893 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znbf\" (UniqueName: \"kubernetes.io/projected/653dc08d-064b-469c-8583-b87821c2d482-kube-api-access-8znbf\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.216111 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.215925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653dc08d-064b-469c-8583-b87821c2d482-tls-certs\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.270294 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.270174 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:10.316426 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.316387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653dc08d-064b-469c-8583-b87821c2d482-tmp\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.316426 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.316425 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znbf\" (UniqueName: \"kubernetes.io/projected/653dc08d-064b-469c-8583-b87821c2d482-kube-api-access-8znbf\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.316649 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.316565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653dc08d-064b-469c-8583-b87821c2d482-tls-certs\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.318764 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.318739 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653dc08d-064b-469c-8583-b87821c2d482-tmp\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.318955 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.318937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653dc08d-064b-469c-8583-b87821c2d482-tls-certs\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.324646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.324612 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znbf\" (UniqueName: \"kubernetes.io/projected/653dc08d-064b-469c-8583-b87821c2d482-kube-api-access-8znbf\") pod \"kube-auth-proxy-6b79c4dfc8-9sg2x\" (UID: \"653dc08d-064b-469c-8583-b87821c2d482\") " pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.471726 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.471688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" Sep 17 01:47:10.763627 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.763598 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x"] Sep 17 01:47:10.766041 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:47:10.766003 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653dc08d_064b_469c_8583_b87821c2d482.slice/crio-c44bff81a776df7eb5bdee0546625cd91f51b93066bea4cbc78782567d25727c WatchSource:0}: Error finding container c44bff81a776df7eb5bdee0546625cd91f51b93066bea4cbc78782567d25727c: Status 404 returned error can't find the container with id c44bff81a776df7eb5bdee0546625cd91f51b93066bea4cbc78782567d25727c Sep 17 01:47:10.869811 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:10.869730 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" event={"ID":"653dc08d-064b-469c-8583-b87821c2d482","Type":"ContainerStarted","Data":"c44bff81a776df7eb5bdee0546625cd91f51b93066bea4cbc78782567d25727c"} Sep 17 01:47:11.353961 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:11.353914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:11.416595 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:11.416494 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:12.337866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:12.337829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:13.886768 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:13.886682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" event={"ID":"653dc08d-064b-469c-8583-b87821c2d482","Type":"ContainerStarted","Data":"458d5678ebac0564630b6dab8db74224e00b81ca0736303358b0a53df1462773"} Sep 17 01:47:13.901027 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:13.900973 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b79c4dfc8-9sg2x" podStartSLOduration=1.175137825 podStartE2EDuration="3.900928243s" totalImagesPullingTime="2.725790418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:13.90045381 +0000 UTC m=+578.735506144" watchObservedRunningTime="2026-09-17 01:47:13.900928243 +0000 UTC m=+578.735980576" Sep 17 01:47:17.119130 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:17.119096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:17.176508 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:17.176484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:17.692324 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:17.692282 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:17.785073 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:17.785041 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:17.890864 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:17.890841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:20.913139 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:20.913097 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:21.016351 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:21.016319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:21.115790 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:21.115766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:21.213918 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:21.213890 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:21.386637 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:21.386612 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:21.687171 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:21.687098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:24.428149 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:24.428101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:24.529507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:24.529468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:29.850744 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:29.850703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:29.945113 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:29.945076 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:30.043937 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:30.043902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:30.327930 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:30.327884 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:31.167920 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:31.167886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:31.266958 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:31.266927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:31.392151 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:31.392127 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:40.719669 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:40.719619 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:40.770902 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:40.770878 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:47:40.824350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:47:40.824324 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:21.885769 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:21.885683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:21.950299 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:21.950272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:37.274928 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:37.274891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:38.268308 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:38.268265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:42.912924 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:42.912878 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:45.880770 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:45.880729 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:48.751423 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:48.751385 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:49.283008 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:49.282976 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:49.891282 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:49.891241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:54.076570 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.076530 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg"] Sep 17 01:48:54.078824 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.078805 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.081595 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.081569 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 01:48:54.081736 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.081580 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n2jw2\"" Sep 17 01:48:54.081859 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.081843 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 01:48:54.105911 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.105882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg"] Sep 17 01:48:54.223978 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.223954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs56x\" (UniqueName: \"kubernetes.io/projected/0195548f-3844-44cc-a42e-b317f98421e3-kube-api-access-qs56x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.224070 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.224018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0195548f-3844-44cc-a42e-b317f98421e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.324380 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.324355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs56x\" (UniqueName: \"kubernetes.io/projected/0195548f-3844-44cc-a42e-b317f98421e3-kube-api-access-qs56x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.324491 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.324414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0195548f-3844-44cc-a42e-b317f98421e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.324747 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.324728 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0195548f-3844-44cc-a42e-b317f98421e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.333175 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.333113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs56x\" (UniqueName: \"kubernetes.io/projected/0195548f-3844-44cc-a42e-b317f98421e3-kube-api-access-qs56x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.389882 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.389859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:48:54.664971 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:54.664946 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg"] Sep 17 01:48:54.667616 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:48:54.667587 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0195548f_3844_44cc_a42e_b317f98421e3.slice/crio-9f43bb9a1223f8b5e93860c77ab6d08e4d49a2417d7964ef44c1acc5890f9bc6 WatchSource:0}: Error finding container 9f43bb9a1223f8b5e93860c77ab6d08e4d49a2417d7964ef44c1acc5890f9bc6: Status 404 returned error can't find the container with id 9f43bb9a1223f8b5e93860c77ab6d08e4d49a2417d7964ef44c1acc5890f9bc6 Sep 17 01:48:55.348098 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:55.348060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" event={"ID":"0195548f-3844-44cc-a42e-b317f98421e3","Type":"ContainerStarted","Data":"9f43bb9a1223f8b5e93860c77ab6d08e4d49a2417d7964ef44c1acc5890f9bc6"} Sep 17 01:48:57.719168 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:57.719132 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:59.169360 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:59.169326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:59.341045 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:59.341008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:48:59.444396 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:48:59.444272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:00.139853 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:00.139817 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:00.948585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:00.948551 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:01.052479 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:01.052445 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:04.077108 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:04.077069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:04.178476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:04.178436 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:05.398790 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:05.398749 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" event={"ID":"0195548f-3844-44cc-a42e-b317f98421e3","Type":"ContainerStarted","Data":"95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255"} Sep 17 01:49:05.399166 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:05.398814 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:05.420706 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:05.420662 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" podStartSLOduration=1.71019686 podStartE2EDuration="11.420645403s" totalImagesPullingTime="9.710448543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:05.418465954 +0000 UTC m=+690.253518293" watchObservedRunningTime="2026-09-17 01:49:05.420645403 +0000 UTC m=+690.255697737" Sep 17 01:49:07.054127 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:07.054092 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:07.433712 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:07.433625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:08.073665 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:08.073620 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:08.127006 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:08.126981 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:08.284695 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:08.284672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:08.384467 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:08.384409 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:08.484687 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:08.484662 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:08.538141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:08.538116 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:15.714202 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:15.714158 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:15.764336 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:15.764305 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:15.905625 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:15.905595 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:16.005838 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:16.005769 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:16.404943 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:16.404872 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:17.461941 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.461911 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp"] Sep 17 01:49:17.465219 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.465193 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.478936 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.478916 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp"] Sep 17 01:49:17.486865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.486843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f3b00f7-3038-46ea-8788-a4f7756718e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.486961 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.486891 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfslk\" (UniqueName: \"kubernetes.io/projected/5f3b00f7-3038-46ea-8788-a4f7756718e3-kube-api-access-xfslk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.520932 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.520899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:17.588027 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.588002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfslk\" (UniqueName: \"kubernetes.io/projected/5f3b00f7-3038-46ea-8788-a4f7756718e3-kube-api-access-xfslk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.588144 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.588057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f3b00f7-3038-46ea-8788-a4f7756718e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.588432 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.588416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f3b00f7-3038-46ea-8788-a4f7756718e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.598340 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.598316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfslk\" (UniqueName: \"kubernetes.io/projected/5f3b00f7-3038-46ea-8788-a4f7756718e3-kube-api-access-xfslk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.619014 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.618987 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:17.720873 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.720818 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:17.775696 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.775667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:17.826317 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:17.826253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:18.064381 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.064347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp"] Sep 17 01:49:18.066814 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:49:18.066786 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3b00f7_3038_46ea_8788_a4f7756718e3.slice/crio-cda5cf2022952f94cb211305fb63e7d48b16aba40f755db909bf507573a89c56 WatchSource:0}: Error finding container cda5cf2022952f94cb211305fb63e7d48b16aba40f755db909bf507573a89c56: Status 404 returned error can't find the container with id cda5cf2022952f94cb211305fb63e7d48b16aba40f755db909bf507573a89c56 Sep 17 01:49:18.458862 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.458785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" event={"ID":"5f3b00f7-3038-46ea-8788-a4f7756718e3","Type":"ContainerStarted","Data":"6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d"} Sep 17 01:49:18.458862 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.458833 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" event={"ID":"5f3b00f7-3038-46ea-8788-a4f7756718e3","Type":"ContainerStarted","Data":"cda5cf2022952f94cb211305fb63e7d48b16aba40f755db909bf507573a89c56"} Sep 17 01:49:18.459027 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.458878 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:18.478832 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.478788 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" podStartSLOduration=1.478774707 podStartE2EDuration="1.478774707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:18.476601438 +0000 UTC m=+703.311653770" watchObservedRunningTime="2026-09-17 01:49:18.478774707 +0000 UTC m=+703.313827097" Sep 17 01:49:18.680841 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.680810 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp"] Sep 17 01:49:18.690910 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.690885 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp"] Sep 17 01:49:18.700724 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.700699 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc"] Sep 17 01:49:18.701016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.701003 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" containerName="manager" Sep 17 01:49:18.701016 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.701016 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" containerName="manager" Sep 17 01:49:18.701135 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.701080 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" containerName="manager" Sep 17 01:49:18.704056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.704034 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.704420 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.704403 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg"] Sep 17 01:49:18.704565 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.704547 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" podUID="0195548f-3844-44cc-a42e-b317f98421e3" containerName="manager" containerID="cri-o://95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255" gracePeriod=2 Sep 17 01:49:18.709271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.708610 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg"] Sep 17 01:49:18.727013 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.726903 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc"] Sep 17 01:49:18.733904 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.733880 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg"] Sep 17 01:49:18.734410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.734392 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0195548f-3844-44cc-a42e-b317f98421e3" containerName="manager" Sep 17 01:49:18.734410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.734411 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0195548f-3844-44cc-a42e-b317f98421e3" containerName="manager" Sep 17 01:49:18.734558 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.734491 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0195548f-3844-44cc-a42e-b317f98421e3" containerName="manager" Sep 17 01:49:18.737677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.737657 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.756267 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.756245 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg"] Sep 17 01:49:18.773838 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:18.773736 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="0195548f-3844-44cc-a42e-b317f98421e3" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:18.796956 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.796934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd200c49-1686-46ad-a6fc-0369fe9154d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-khcjg\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.797059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.796967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8033edd-5fe7-4cec-8d02-277e1ce30415-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kkwlc\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.797059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.796993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pms\" (UniqueName: \"kubernetes.io/projected/bd200c49-1686-46ad-a6fc-0369fe9154d3-kube-api-access-r6pms\") pod \"kuadrant-operator-controller-manager-76f8f686bf-khcjg\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.797059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.797018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfshq\" (UniqueName: \"kubernetes.io/projected/f8033edd-5fe7-4cec-8d02-277e1ce30415-kube-api-access-zfshq\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kkwlc\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.897530 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.897497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd200c49-1686-46ad-a6fc-0369fe9154d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-khcjg\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.897686 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.897552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8033edd-5fe7-4cec-8d02-277e1ce30415-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kkwlc\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.897686 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.897593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pms\" (UniqueName: \"kubernetes.io/projected/bd200c49-1686-46ad-a6fc-0369fe9154d3-kube-api-access-r6pms\") pod \"kuadrant-operator-controller-manager-76f8f686bf-khcjg\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.897686 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.897621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfshq\" (UniqueName: \"kubernetes.io/projected/f8033edd-5fe7-4cec-8d02-277e1ce30415-kube-api-access-zfshq\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kkwlc\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.897862 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.897841 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd200c49-1686-46ad-a6fc-0369fe9154d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-khcjg\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.897927 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.897910 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8033edd-5fe7-4cec-8d02-277e1ce30415-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kkwlc\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.909117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.909097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pms\" (UniqueName: \"kubernetes.io/projected/bd200c49-1686-46ad-a6fc-0369fe9154d3-kube-api-access-r6pms\") pod \"kuadrant-operator-controller-manager-76f8f686bf-khcjg\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:18.909285 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.909229 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfshq\" (UniqueName: \"kubernetes.io/projected/f8033edd-5fe7-4cec-8d02-277e1ce30415-kube-api-access-zfshq\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-kkwlc\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:18.930794 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.930775 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:18.932770 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:18.932746 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="0195548f-3844-44cc-a42e-b317f98421e3" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:18.998190 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.998170 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs56x\" (UniqueName: \"kubernetes.io/projected/0195548f-3844-44cc-a42e-b317f98421e3-kube-api-access-qs56x\") pod \"0195548f-3844-44cc-a42e-b317f98421e3\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " Sep 17 01:49:18.998300 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.998274 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0195548f-3844-44cc-a42e-b317f98421e3-extensions-socket-volume\") pod \"0195548f-3844-44cc-a42e-b317f98421e3\" (UID: \"0195548f-3844-44cc-a42e-b317f98421e3\") " Sep 17 01:49:18.998725 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:18.998701 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0195548f-3844-44cc-a42e-b317f98421e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0195548f-3844-44cc-a42e-b317f98421e3" (UID: "0195548f-3844-44cc-a42e-b317f98421e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:19.000077 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.000059 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0195548f-3844-44cc-a42e-b317f98421e3-kube-api-access-qs56x" (OuterVolumeSpecName: "kube-api-access-qs56x") pod "0195548f-3844-44cc-a42e-b317f98421e3" (UID: "0195548f-3844-44cc-a42e-b317f98421e3"). InnerVolumeSpecName "kube-api-access-qs56x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:19.018256 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.018234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:19.077613 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.077584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:19.099369 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.099345 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0195548f-3844-44cc-a42e-b317f98421e3-extensions-socket-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:19.099369 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.099369 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs56x\" (UniqueName: \"kubernetes.io/projected/0195548f-3844-44cc-a42e-b317f98421e3-kube-api-access-qs56x\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:19.293974 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.293915 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc"] Sep 17 01:49:19.297242 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:49:19.297196 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8033edd_5fe7_4cec_8d02_277e1ce30415.slice/crio-c8ef30f3fdcb55a66379efdd2fbda334837f59fd7d24bd5a4cf41e8540697ade WatchSource:0}: Error finding container c8ef30f3fdcb55a66379efdd2fbda334837f59fd7d24bd5a4cf41e8540697ade: Status 404 returned error can't find the container with id c8ef30f3fdcb55a66379efdd2fbda334837f59fd7d24bd5a4cf41e8540697ade Sep 17 01:49:19.350755 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.350734 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg"] Sep 17 01:49:19.353906 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:49:19.353877 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd200c49_1686_46ad_a6fc_0369fe9154d3.slice/crio-9c8c4e6ba1cc315ab05da058775cc350efd7fb3d669a681c8b8ccf06bc785c23 WatchSource:0}: Error finding container 9c8c4e6ba1cc315ab05da058775cc350efd7fb3d669a681c8b8ccf06bc785c23: Status 404 returned error can't find the container with id 9c8c4e6ba1cc315ab05da058775cc350efd7fb3d669a681c8b8ccf06bc785c23 Sep 17 01:49:19.465701 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.465664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" event={"ID":"bd200c49-1686-46ad-a6fc-0369fe9154d3","Type":"ContainerStarted","Data":"b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc"} Sep 17 01:49:19.465839 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.465724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" event={"ID":"bd200c49-1686-46ad-a6fc-0369fe9154d3","Type":"ContainerStarted","Data":"9c8c4e6ba1cc315ab05da058775cc350efd7fb3d669a681c8b8ccf06bc785c23"} Sep 17 01:49:19.465891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.465844 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:19.467130 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.467109 2582 generic.go:356] "Generic (PLEG): container finished" podID="0195548f-3844-44cc-a42e-b317f98421e3" containerID="95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255" exitCode=0 Sep 17 01:49:19.467243 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.467147 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:19.467243 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.467217 2582 scope.go:117] "RemoveContainer" containerID="95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255" Sep 17 01:49:19.469056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.469030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" event={"ID":"f8033edd-5fe7-4cec-8d02-277e1ce30415","Type":"ContainerStarted","Data":"3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10"} Sep 17 01:49:19.469173 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.469069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" event={"ID":"f8033edd-5fe7-4cec-8d02-277e1ce30415","Type":"ContainerStarted","Data":"c8ef30f3fdcb55a66379efdd2fbda334837f59fd7d24bd5a4cf41e8540697ade"} Sep 17 01:49:19.469333 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:19.469312 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" container="manager" Sep 17 01:49:19.475933 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.475918 2582 scope.go:117] "RemoveContainer" containerID="95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255" Sep 17 01:49:19.476190 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:19.476174 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255\": container with ID starting with 95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255 not found: ID does not exist" containerID="95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255" Sep 17 01:49:19.476268 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.476197 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255"} err="failed to get container status \"95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255\": rpc error: code = NotFound desc = could not find container \"95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255\": container with ID starting with 95aa4e74f740a11a3c6260ca13f4d833d3c0afbc9ae2b4dcdcbb56a9e1a80255 not found: ID does not exist" Sep 17 01:49:19.493453 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.493415 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" podStartSLOduration=1.493400189 podStartE2EDuration="1.493400189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:19.493157058 +0000 UTC m=+704.328209392" watchObservedRunningTime="2026-09-17 01:49:19.493400189 +0000 UTC m=+704.328452523" Sep 17 01:49:19.494995 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:19.494974 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="0195548f-3844-44cc-a42e-b317f98421e3" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:19.496672 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:19.496653 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5jmzg\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="0195548f-3844-44cc-a42e-b317f98421e3" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5jmzg" Sep 17 01:49:19.498229 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:19.498201 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:19.679432 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:19.679352 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0195548f-3844-44cc-a42e-b317f98421e3" path="/var/lib/kubelet/pods/0195548f-3844-44cc-a42e-b317f98421e3/volumes" Sep 17 01:49:20.476150 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.476107 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" containerName="manager" containerID="cri-o://6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d" gracePeriod=2 Sep 17 01:49:20.477430 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.477398 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:20.711404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.711379 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:20.812673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.812622 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f3b00f7-3038-46ea-8788-a4f7756718e3-extensions-socket-volume\") pod \"5f3b00f7-3038-46ea-8788-a4f7756718e3\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " Sep 17 01:49:20.812673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.812658 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfslk\" (UniqueName: \"kubernetes.io/projected/5f3b00f7-3038-46ea-8788-a4f7756718e3-kube-api-access-xfslk\") pod \"5f3b00f7-3038-46ea-8788-a4f7756718e3\" (UID: \"5f3b00f7-3038-46ea-8788-a4f7756718e3\") " Sep 17 01:49:20.812973 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.812949 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3b00f7-3038-46ea-8788-a4f7756718e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5f3b00f7-3038-46ea-8788-a4f7756718e3" (UID: "5f3b00f7-3038-46ea-8788-a4f7756718e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:20.814621 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.814600 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3b00f7-3038-46ea-8788-a4f7756718e3-kube-api-access-xfslk" (OuterVolumeSpecName: "kube-api-access-xfslk") pod "5f3b00f7-3038-46ea-8788-a4f7756718e3" (UID: "5f3b00f7-3038-46ea-8788-a4f7756718e3"). InnerVolumeSpecName "kube-api-access-xfslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:20.914083 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.914058 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f3b00f7-3038-46ea-8788-a4f7756718e3-extensions-socket-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:20.914083 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:20.914083 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfslk\" (UniqueName: \"kubernetes.io/projected/5f3b00f7-3038-46ea-8788-a4f7756718e3-kube-api-access-xfslk\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:21.260102 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.260074 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:21.483614 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.483585 2582 generic.go:356] "Generic (PLEG): container finished" podID="5f3b00f7-3038-46ea-8788-a4f7756718e3" containerID="6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d" exitCode=0 Sep 17 01:49:21.483719 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.483637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:21.483719 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.483690 2582 scope.go:117] "RemoveContainer" containerID="6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d" Sep 17 01:49:21.485881 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:21.485845 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:21.493009 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.492988 2582 scope.go:117] "RemoveContainer" containerID="6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d" Sep 17 01:49:21.493350 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:21.493329 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d\": container with ID starting with 6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d not found: ID does not exist" containerID="6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d" Sep 17 01:49:21.493416 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.493358 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d"} err="failed to get container status \"6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d\": rpc error: code = NotFound desc = could not find container \"6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d\": container with ID starting with 6ae3ea83a1329046cc28d5cbfd6c9b4874084f4282ce0ca271f1d6a4192e1b2d not found: ID does not exist" Sep 17 01:49:21.494132 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:21.494111 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-62bdp\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62bdp" Sep 17 01:49:21.679567 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:21.679514 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3b00f7-3038-46ea-8788-a4f7756718e3" path="/var/lib/kubelet/pods/5f3b00f7-3038-46ea-8788-a4f7756718e3/volumes" Sep 17 01:49:22.065722 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:22.065680 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:22.195306 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:22.195272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:26.799492 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:26.799447 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:26.896396 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:26.896337 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:26.996308 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:26.996267 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:30.479832 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:30.479750 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:30.501461 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:30.501411 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" podStartSLOduration=12.5013941 podStartE2EDuration="12.5013941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:19.518773724 +0000 UTC m=+704.353826058" watchObservedRunningTime="2026-09-17 01:49:30.5013941 +0000 UTC m=+715.336446434" Sep 17 01:49:31.486614 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.486587 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:31.545438 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.545410 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg"] Sep 17 01:49:31.545600 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.545573 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" podUID="bd200c49-1686-46ad-a6fc-0369fe9154d3" containerName="manager" containerID="cri-o://b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc" gracePeriod=10 Sep 17 01:49:31.740169 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.740110 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd"] Sep 17 01:49:31.756784 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.756760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.758103 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.758080 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd"] Sep 17 01:49:31.791840 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.791815 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14decd50-2064-4ccc-ac01-1ddeaba37b4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6mnsd\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.791982 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.791965 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26s7\" (UniqueName: \"kubernetes.io/projected/14decd50-2064-4ccc-ac01-1ddeaba37b4b-kube-api-access-v26s7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6mnsd\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.816427 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.816407 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:31.893058 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.893032 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6pms\" (UniqueName: \"kubernetes.io/projected/bd200c49-1686-46ad-a6fc-0369fe9154d3-kube-api-access-r6pms\") pod \"bd200c49-1686-46ad-a6fc-0369fe9154d3\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " Sep 17 01:49:31.893173 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.893092 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd200c49-1686-46ad-a6fc-0369fe9154d3-extensions-socket-volume\") pod \"bd200c49-1686-46ad-a6fc-0369fe9154d3\" (UID: \"bd200c49-1686-46ad-a6fc-0369fe9154d3\") " Sep 17 01:49:31.893300 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.893281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14decd50-2064-4ccc-ac01-1ddeaba37b4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6mnsd\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.893374 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.893354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26s7\" (UniqueName: \"kubernetes.io/projected/14decd50-2064-4ccc-ac01-1ddeaba37b4b-kube-api-access-v26s7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6mnsd\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.893428 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.893394 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd200c49-1686-46ad-a6fc-0369fe9154d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bd200c49-1686-46ad-a6fc-0369fe9154d3" (UID: "bd200c49-1686-46ad-a6fc-0369fe9154d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:31.893619 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.893588 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14decd50-2064-4ccc-ac01-1ddeaba37b4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6mnsd\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.894974 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.894954 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd200c49-1686-46ad-a6fc-0369fe9154d3-kube-api-access-r6pms" (OuterVolumeSpecName: "kube-api-access-r6pms") pod "bd200c49-1686-46ad-a6fc-0369fe9154d3" (UID: "bd200c49-1686-46ad-a6fc-0369fe9154d3"). InnerVolumeSpecName "kube-api-access-r6pms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:31.903506 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.903486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26s7\" (UniqueName: \"kubernetes.io/projected/14decd50-2064-4ccc-ac01-1ddeaba37b4b-kube-api-access-v26s7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6mnsd\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:31.994527 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.994475 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pms\" (UniqueName: \"kubernetes.io/projected/bd200c49-1686-46ad-a6fc-0369fe9154d3-kube-api-access-r6pms\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:31.994527 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:31.994497 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd200c49-1686-46ad-a6fc-0369fe9154d3-extensions-socket-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:32.069627 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.069604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:32.384434 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.384411 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd"] Sep 17 01:49:32.386542 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:49:32.386515 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14decd50_2064_4ccc_ac01_1ddeaba37b4b.slice/crio-3d5718676b8c1a8469af6beaad4619a9d8e6ba6e206db5e42391c200dc7e5e0c WatchSource:0}: Error finding container 3d5718676b8c1a8469af6beaad4619a9d8e6ba6e206db5e42391c200dc7e5e0c: Status 404 returned error can't find the container with id 3d5718676b8c1a8469af6beaad4619a9d8e6ba6e206db5e42391c200dc7e5e0c Sep 17 01:49:32.540466 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.540434 2582 generic.go:356] "Generic (PLEG): container finished" podID="bd200c49-1686-46ad-a6fc-0369fe9154d3" containerID="b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc" exitCode=0 Sep 17 01:49:32.540792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.540472 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" Sep 17 01:49:32.540792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.540519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" event={"ID":"bd200c49-1686-46ad-a6fc-0369fe9154d3","Type":"ContainerDied","Data":"b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc"} Sep 17 01:49:32.540792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.540576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg" event={"ID":"bd200c49-1686-46ad-a6fc-0369fe9154d3","Type":"ContainerDied","Data":"9c8c4e6ba1cc315ab05da058775cc350efd7fb3d669a681c8b8ccf06bc785c23"} Sep 17 01:49:32.540792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.540607 2582 scope.go:117] "RemoveContainer" containerID="b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc" Sep 17 01:49:32.542415 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.542390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" event={"ID":"14decd50-2064-4ccc-ac01-1ddeaba37b4b","Type":"ContainerStarted","Data":"317bb5dbfcd1e33c8a1ec83c81f1c0f1928c945e6c2c27450c86edcd957d97c5"} Sep 17 01:49:32.542522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.542436 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" event={"ID":"14decd50-2064-4ccc-ac01-1ddeaba37b4b","Type":"ContainerStarted","Data":"3d5718676b8c1a8469af6beaad4619a9d8e6ba6e206db5e42391c200dc7e5e0c"} Sep 17 01:49:32.542522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.542464 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:32.548685 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.548510 2582 scope.go:117] "RemoveContainer" containerID="b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc" Sep 17 01:49:32.548802 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:32.548783 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc\": container with ID starting with b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc not found: ID does not exist" containerID="b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc" Sep 17 01:49:32.548846 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.548811 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc"} err="failed to get container status \"b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc\": rpc error: code = NotFound desc = could not find container \"b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc\": container with ID starting with b23e7be7233a7c127e1e93181dd6a8d3d3c93b4d4dae1c29b82aa3e2ce0ae5dc not found: ID does not exist" Sep 17 01:49:32.572096 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.572059 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" podStartSLOduration=1.5720465099999998 podStartE2EDuration="1.57204651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:49:32.570952104 +0000 UTC m=+717.406004437" watchObservedRunningTime="2026-09-17 01:49:32.57204651 +0000 UTC m=+717.407098844" Sep 17 01:49:32.594024 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.593998 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg"] Sep 17 01:49:32.599557 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:32.599538 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-khcjg"] Sep 17 01:49:33.678733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:33.678692 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd200c49-1686-46ad-a6fc-0369fe9154d3" path="/var/lib/kubelet/pods/bd200c49-1686-46ad-a6fc-0369fe9154d3/volumes" Sep 17 01:49:34.627081 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:34.627038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:34.685260 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:34.685229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:35.225947 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:35.225914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:35.326099 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:35.326073 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:37.622031 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:37.621994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:37.678645 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:37.678603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:42.826070 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:42.826032 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:42.927167 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:42.927125 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:43.550869 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.550836 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 01:49:43.605819 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.605791 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc"] Sep 17 01:49:43.606006 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.605984 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" podUID="f8033edd-5fe7-4cec-8d02-277e1ce30415" containerName="manager" containerID="cri-o://3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10" gracePeriod=10 Sep 17 01:49:43.848107 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.848085 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:43.978135 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.978111 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfshq\" (UniqueName: \"kubernetes.io/projected/f8033edd-5fe7-4cec-8d02-277e1ce30415-kube-api-access-zfshq\") pod \"f8033edd-5fe7-4cec-8d02-277e1ce30415\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " Sep 17 01:49:43.978295 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.978176 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8033edd-5fe7-4cec-8d02-277e1ce30415-extensions-socket-volume\") pod \"f8033edd-5fe7-4cec-8d02-277e1ce30415\" (UID: \"f8033edd-5fe7-4cec-8d02-277e1ce30415\") " Sep 17 01:49:43.978569 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.978545 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8033edd-5fe7-4cec-8d02-277e1ce30415-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f8033edd-5fe7-4cec-8d02-277e1ce30415" (UID: "f8033edd-5fe7-4cec-8d02-277e1ce30415"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:43.980044 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:43.980025 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8033edd-5fe7-4cec-8d02-277e1ce30415-kube-api-access-zfshq" (OuterVolumeSpecName: "kube-api-access-zfshq") pod "f8033edd-5fe7-4cec-8d02-277e1ce30415" (UID: "f8033edd-5fe7-4cec-8d02-277e1ce30415"). InnerVolumeSpecName "kube-api-access-zfshq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:44.078792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.078758 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfshq\" (UniqueName: \"kubernetes.io/projected/f8033edd-5fe7-4cec-8d02-277e1ce30415-kube-api-access-zfshq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:44.078792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.078788 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8033edd-5fe7-4cec-8d02-277e1ce30415-extensions-socket-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:49:44.602121 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.602093 2582 generic.go:356] "Generic (PLEG): container finished" podID="f8033edd-5fe7-4cec-8d02-277e1ce30415" containerID="3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10" exitCode=0 Sep 17 01:49:44.602261 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.602142 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" Sep 17 01:49:44.602261 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.602198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" event={"ID":"f8033edd-5fe7-4cec-8d02-277e1ce30415","Type":"ContainerDied","Data":"3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10"} Sep 17 01:49:44.602343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.602282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc" event={"ID":"f8033edd-5fe7-4cec-8d02-277e1ce30415","Type":"ContainerDied","Data":"c8ef30f3fdcb55a66379efdd2fbda334837f59fd7d24bd5a4cf41e8540697ade"} Sep 17 01:49:44.602343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.602310 2582 scope.go:117] "RemoveContainer" containerID="3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10" Sep 17 01:49:44.612025 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.611850 2582 scope.go:117] "RemoveContainer" containerID="3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10" Sep 17 01:49:44.612177 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:49:44.612156 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10\": container with ID starting with 3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10 not found: ID does not exist" containerID="3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10" Sep 17 01:49:44.612284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.612190 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10"} err="failed to get container status \"3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10\": rpc error: code = NotFound desc = could not find container \"3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10\": container with ID starting with 3f29c765db1f636dbbfc1105ce99b433b73da6be18c27e3bd6a092f0f987af10 not found: ID does not exist" Sep 17 01:49:44.626841 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.626820 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc"] Sep 17 01:49:44.635041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:44.635008 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-kkwlc"] Sep 17 01:49:45.678265 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:45.678232 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8033edd-5fe7-4cec-8d02-277e1ce30415" path="/var/lib/kubelet/pods/f8033edd-5fe7-4cec-8d02-277e1ce30415/volumes" Sep 17 01:49:46.571907 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:46.571874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:46.671038 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:46.671008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:46.771966 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:46.771936 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:47.930865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:47.930829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:48.140446 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:48.140412 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:59.666263 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:59.666226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:59.777384 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:59.777354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:49:59.837380 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:49:59.837347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:00.083999 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.083974 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:50:00.084294 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.084281 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8033edd-5fe7-4cec-8d02-277e1ce30415" containerName="manager" Sep 17 01:50:00.084357 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.084296 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8033edd-5fe7-4cec-8d02-277e1ce30415" containerName="manager" Sep 17 01:50:00.084357 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.084307 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd200c49-1686-46ad-a6fc-0369fe9154d3" containerName="manager" Sep 17 01:50:00.084357 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.084312 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd200c49-1686-46ad-a6fc-0369fe9154d3" containerName="manager" Sep 17 01:50:00.084460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.084373 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd200c49-1686-46ad-a6fc-0369fe9154d3" containerName="manager" Sep 17 01:50:00.084460 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.084384 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8033edd-5fe7-4cec-8d02-277e1ce30415" containerName="manager" Sep 17 01:50:00.087249 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.087232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.089652 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.089637 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 17 01:50:00.089713 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.089640 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6khtx\"" Sep 17 01:50:00.095896 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.095874 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:50:00.131707 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.131686 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:50:00.196855 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.196834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/100eb694-9a72-4066-a9d4-2b4dce3e430c-config-file\") pod \"limitador-limitador-84994f7ffc-wjsqq\" (UID: \"100eb694-9a72-4066-a9d4-2b4dce3e430c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.196938 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.196886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tv4\" (UniqueName: \"kubernetes.io/projected/100eb694-9a72-4066-a9d4-2b4dce3e430c-kube-api-access-v5tv4\") pod \"limitador-limitador-84994f7ffc-wjsqq\" (UID: \"100eb694-9a72-4066-a9d4-2b4dce3e430c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.298089 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.298065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tv4\" (UniqueName: \"kubernetes.io/projected/100eb694-9a72-4066-a9d4-2b4dce3e430c-kube-api-access-v5tv4\") pod \"limitador-limitador-84994f7ffc-wjsqq\" (UID: \"100eb694-9a72-4066-a9d4-2b4dce3e430c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.298202 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.298135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/100eb694-9a72-4066-a9d4-2b4dce3e430c-config-file\") pod \"limitador-limitador-84994f7ffc-wjsqq\" (UID: \"100eb694-9a72-4066-a9d4-2b4dce3e430c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.298695 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.298677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/100eb694-9a72-4066-a9d4-2b4dce3e430c-config-file\") pod \"limitador-limitador-84994f7ffc-wjsqq\" (UID: \"100eb694-9a72-4066-a9d4-2b4dce3e430c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.305756 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.305732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tv4\" (UniqueName: \"kubernetes.io/projected/100eb694-9a72-4066-a9d4-2b4dce3e430c-kube-api-access-v5tv4\") pod \"limitador-limitador-84994f7ffc-wjsqq\" (UID: \"100eb694-9a72-4066-a9d4-2b4dce3e430c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.398733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.398674 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:00.503013 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.502985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:00.591321 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.591288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:00.664528 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.664505 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:50:00.666815 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:50:00.666790 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100eb694_9a72_4066_a9d4_2b4dce3e430c.slice/crio-e0208d157cdbba0707d5f5969f71f2e43f80c539ade0fedb18ca02e1b6dffa4a WatchSource:0}: Error finding container e0208d157cdbba0707d5f5969f71f2e43f80c539ade0fedb18ca02e1b6dffa4a: Status 404 returned error can't find the container with id e0208d157cdbba0707d5f5969f71f2e43f80c539ade0fedb18ca02e1b6dffa4a Sep 17 01:50:00.676347 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:00.676323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" event={"ID":"100eb694-9a72-4066-a9d4-2b4dce3e430c","Type":"ContainerStarted","Data":"e0208d157cdbba0707d5f5969f71f2e43f80c539ade0fedb18ca02e1b6dffa4a"} Sep 17 01:50:01.682804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:01.682767 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" event={"ID":"100eb694-9a72-4066-a9d4-2b4dce3e430c","Type":"ContainerStarted","Data":"4298783b9e30b7ce0d31969d2c01c636872a396f68fc3008d012551c3e4a4632"} Sep 17 01:50:01.683180 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:01.682899 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:01.699393 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:01.699350 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" podStartSLOduration=0.860535493 podStartE2EDuration="1.69933726s" totalImagesPullingTime="838.801767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:01.69835245 +0000 UTC m=+746.533404785" watchObservedRunningTime="2026-09-17 01:50:01.69933726 +0000 UTC m=+746.534389593" Sep 17 01:50:12.688944 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:12.688906 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wjsqq" Sep 17 01:50:16.504390 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:16.504352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:16.607337 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:16.607302 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:16.984388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:16.984358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:29.635887 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:29.635854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:29.774554 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:29.774521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:36.215502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:36.215465 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:36.314538 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:36.314503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:36.401806 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:36.401774 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:36.632564 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:36.632490 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:39.744099 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:39.744067 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:50:39.839088 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:50:39.839061 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:07.736317 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:07.736199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:11.308029 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:11.307993 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:11.413104 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:11.413069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:12.810844 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:12.810806 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:12.910193 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:12.910154 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:13.061256 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:13.061139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:13.289392 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:13.289353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:13.391664 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:13.391564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:13.480081 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:13.480043 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:51:13.690666 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:13.690577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:24.584887 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:24.584849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:24.683031 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:24.682994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:34.691897 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:34.691861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:35.102990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:35.102956 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:51:41.881129 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:41.881093 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:42.505115 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:42.505077 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:51:54.123635 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:54.123600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:51:54.872493 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:51:54.872463 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:52:01.596883 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:01.596852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:52:03.868178 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.868148 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh"] Sep 17 01:52:03.871445 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.871422 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:03.874671 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.874641 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k5l9j\"" Sep 17 01:52:03.874802 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.874671 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 01:52:03.874802 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.874642 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 17 01:52:03.874933 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.874813 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 01:52:03.883201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.883179 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh"] Sep 17 01:52:03.895815 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.895787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:52:03.936918 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.936897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:03.937036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.936938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdgh\" (UniqueName: \"kubernetes.io/projected/2a65e674-39df-4070-90a5-b13cb17f3e82-kube-api-access-hbdgh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:03.937036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.936988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:03.937036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.937022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:03.937195 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.937102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a65e674-39df-4070-90a5-b13cb17f3e82-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:03.937195 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:03.937138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038298 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdgh\" (UniqueName: \"kubernetes.io/projected/2a65e674-39df-4070-90a5-b13cb17f3e82-kube-api-access-hbdgh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038425 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038425 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038425 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a65e674-39df-4070-90a5-b13cb17f3e82-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038425 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038765 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038765 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038880 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.038880 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.038834 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.040616 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.040592 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a65e674-39df-4070-90a5-b13cb17f3e82-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.040771 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.040753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a65e674-39df-4070-90a5-b13cb17f3e82-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.046358 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.046340 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdgh\" (UniqueName: \"kubernetes.io/projected/2a65e674-39df-4070-90a5-b13cb17f3e82-kube-api-access-hbdgh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v6qkh\" (UID: \"2a65e674-39df-4070-90a5-b13cb17f3e82\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.183164 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.183106 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:04.448992 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.448970 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh"] Sep 17 01:52:04.451350 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:52:04.451323 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a65e674_39df_4070_90a5_b13cb17f3e82.slice/crio-7176a2482529f341f91fe599c3c4845d90fa59adc8ab301ba93802eb0a6ef89e WatchSource:0}: Error finding container 7176a2482529f341f91fe599c3c4845d90fa59adc8ab301ba93802eb0a6ef89e: Status 404 returned error can't find the container with id 7176a2482529f341f91fe599c3c4845d90fa59adc8ab301ba93802eb0a6ef89e Sep 17 01:52:04.453059 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.453040 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:52:04.565967 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:04.565939 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:52:05.239486 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:05.239444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" event={"ID":"2a65e674-39df-4070-90a5-b13cb17f3e82","Type":"ContainerStarted","Data":"7176a2482529f341f91fe599c3c4845d90fa59adc8ab301ba93802eb0a6ef89e"} Sep 17 01:52:06.245527 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.245496 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" event={"ID":"2a65e674-39df-4070-90a5-b13cb17f3e82","Type":"ContainerStarted","Data":"b2907ef77785e4be4412e269b810b74138920c0912e021eb4ffd191374df54f4"} Sep 17 01:52:06.262219 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.262172 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" podStartSLOduration=2.251424648 podStartE2EDuration="3.262157122s" totalImagesPullingTime="1.010732474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:06.260895085 +0000 UTC m=+871.095947416" watchObservedRunningTime="2026-09-17 01:52:06.262157122 +0000 UTC m=+871.097209457" Sep 17 01:52:06.677693 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.677615 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn"] Sep 17 01:52:06.680969 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.680945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.683300 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.683277 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 17 01:52:06.691644 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.691621 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn"] Sep 17 01:52:06.706055 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.706028 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:52:06.762814 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.762787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnwb\" (UniqueName: \"kubernetes.io/projected/761655a8-e8b6-42ef-9143-0bbafaa9c736-kube-api-access-zcnwb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.762905 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.762823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.762905 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.762844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.762975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.762915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.762975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.762948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761655a8-e8b6-42ef-9143-0bbafaa9c736-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.763084 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.763014 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863577 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863667 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863667 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863667 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761655a8-e8b6-42ef-9143-0bbafaa9c736-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863667 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863831 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnwb\" (UniqueName: \"kubernetes.io/projected/761655a8-e8b6-42ef-9143-0bbafaa9c736-kube-api-access-zcnwb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.863959 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.863941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.864042 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.864002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.864133 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.864112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.865942 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.865923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761655a8-e8b6-42ef-9143-0bbafaa9c736-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.866155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.866135 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761655a8-e8b6-42ef-9143-0bbafaa9c736-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.871618 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.871598 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnwb\" (UniqueName: \"kubernetes.io/projected/761655a8-e8b6-42ef-9143-0bbafaa9c736-kube-api-access-zcnwb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn\" (UID: \"761655a8-e8b6-42ef-9143-0bbafaa9c736\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:06.991658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:06.991632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:07.259086 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:07.259064 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn"] Sep 17 01:52:07.260475 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:52:07.260449 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761655a8_e8b6_42ef_9143_0bbafaa9c736.slice/crio-05059da6b0527b48a5ede597f6e8f4fa71382210bf51b7659defbe87893b5627 WatchSource:0}: Error finding container 05059da6b0527b48a5ede597f6e8f4fa71382210bf51b7659defbe87893b5627: Status 404 returned error can't find the container with id 05059da6b0527b48a5ede597f6e8f4fa71382210bf51b7659defbe87893b5627 Sep 17 01:52:08.255741 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:08.255706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" event={"ID":"761655a8-e8b6-42ef-9143-0bbafaa9c736","Type":"ContainerStarted","Data":"eee4da8b83dd988fb06ca88006fe6d7d11d91e229e1e1bbe3237d5e5b1f88d20"} Sep 17 01:52:08.255906 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:08.255755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" event={"ID":"761655a8-e8b6-42ef-9143-0bbafaa9c736","Type":"ContainerStarted","Data":"05059da6b0527b48a5ede597f6e8f4fa71382210bf51b7659defbe87893b5627"} Sep 17 01:52:08.272163 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:08.272117 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" podStartSLOduration=2.085220108 podStartE2EDuration="2.272101632s" totalImagesPullingTime="186.881524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:08.27087323 +0000 UTC m=+873.105925590" watchObservedRunningTime="2026-09-17 01:52:08.272101632 +0000 UTC m=+873.107153965" Sep 17 01:52:10.879008 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:10.878973 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:52:12.500750 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:12.500716 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:52:14.183530 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:14.183486 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:14.184026 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:14.183702 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:14.196857 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:14.196826 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:14.294107 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:14.294077 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v6qkh" Sep 17 01:52:16.992140 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:16.992100 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:16.992592 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:16.992159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:17.004925 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:17.004896 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:17.306526 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:17.306431 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbg9xn" Sep 17 01:52:17.972663 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:17.972627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:52:18.567928 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.567900 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45"] Sep 17 01:52:18.571440 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.571413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.573843 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.573821 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 17 01:52:18.581601 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.581580 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45"] Sep 17 01:52:18.600114 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.600082 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:52:18.662703 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.662673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99d7765-924f-4509-b438-39927ac715b8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.662703 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.662702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq8k\" (UniqueName: \"kubernetes.io/projected/a99d7765-924f-4509-b438-39927ac715b8-kube-api-access-chq8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.662866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.662729 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.662866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.662765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.662866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.662816 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.662966 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.662878 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.763754 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.763717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.763914 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.763894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.763982 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.763954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99d7765-924f-4509-b438-39927ac715b8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.764036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.763981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq8k\" (UniqueName: \"kubernetes.io/projected/a99d7765-924f-4509-b438-39927ac715b8-kube-api-access-chq8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.764036 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.764027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.764141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.764052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.764141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.764114 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.764290 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.764241 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.764472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.764447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.766261 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.766240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99d7765-924f-4509-b438-39927ac715b8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.766384 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.766363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99d7765-924f-4509-b438-39927ac715b8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.771158 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.771141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq8k\" (UniqueName: \"kubernetes.io/projected/a99d7765-924f-4509-b438-39927ac715b8-kube-api-access-chq8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9tb45\" (UID: \"a99d7765-924f-4509-b438-39927ac715b8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:18.884102 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:18.884035 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:19.156128 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:19.156101 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45"] Sep 17 01:52:19.157705 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:52:19.157676 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda99d7765_924f_4509_b438_39927ac715b8.slice/crio-0eb52823beb65e8e059df9994858ced2e1a548ca7ff3dcc5b16fa97ca7627c76 WatchSource:0}: Error finding container 0eb52823beb65e8e059df9994858ced2e1a548ca7ff3dcc5b16fa97ca7627c76: Status 404 returned error can't find the container with id 0eb52823beb65e8e059df9994858ced2e1a548ca7ff3dcc5b16fa97ca7627c76 Sep 17 01:52:19.306930 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:19.306897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" event={"ID":"a99d7765-924f-4509-b438-39927ac715b8","Type":"ContainerStarted","Data":"0eb52823beb65e8e059df9994858ced2e1a548ca7ff3dcc5b16fa97ca7627c76"} Sep 17 01:52:20.312454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:20.312421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" event={"ID":"a99d7765-924f-4509-b438-39927ac715b8","Type":"ContainerStarted","Data":"6a5ae2efde4cf6126f7aa2f765057d4d2c989c4d0abe90fbee4964d1eaa5c871"} Sep 17 01:52:20.331521 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:20.331474 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" podStartSLOduration=2.147283215 podStartE2EDuration="2.331458572s" totalImagesPullingTime="184.175357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:52:20.329408236 +0000 UTC m=+885.164460570" watchObservedRunningTime="2026-09-17 01:52:20.331458572 +0000 UTC m=+885.166510904" Sep 17 01:52:24.274935 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:24.274902 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:52:28.884710 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:28.884634 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:28.884710 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:28.884683 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:28.897549 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:28.897524 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:29.373463 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:29.373428 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9tb45" Sep 17 01:52:53.117077 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:52:53.117038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:53:04.891714 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:53:04.891674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:53:04.998592 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:53:04.998551 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:53:53.933992 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:53:53.933961 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:53:54.057162 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:53:54.057117 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:53:54.169966 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:53:54.169935 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:02.764119 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:02.764017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:02.816488 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:02.816453 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:11.896976 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:11.896941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:11.970508 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:11.970470 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:27.926799 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:27.926763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:42.301543 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:42.301500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:54:42.677037 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:54:42.676945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:09.593658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:09.593617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:09.694005 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:09.693974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:10.807413 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:10.807379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:10.902477 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:10.902448 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.071991 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.071951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.172570 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.172537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.416072 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.415968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.492382 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.492352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.570560 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.570527 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.670544 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.670466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:27.901006 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:27.900934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:28.588934 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:28.588896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:28.688581 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:28.688549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:29.191285 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.191253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:29.496982 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.496945 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10810-1c0595"] Sep 17 01:55:29.500313 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.500296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:29.502843 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.502818 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7c2dt\"" Sep 17 01:55:29.503895 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.503874 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10810-1c0595"] Sep 17 01:55:29.617569 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.617550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czp5m\" (UniqueName: \"kubernetes.io/projected/13702528-9a6f-4a00-8871-053cbe8163e7-kube-api-access-czp5m\") pod \"test-curl-10810-1c0595\" (UID: \"13702528-9a6f-4a00-8871-053cbe8163e7\") " pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:29.718698 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.718679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czp5m\" (UniqueName: \"kubernetes.io/projected/13702528-9a6f-4a00-8871-053cbe8163e7-kube-api-access-czp5m\") pod \"test-curl-10810-1c0595\" (UID: \"13702528-9a6f-4a00-8871-053cbe8163e7\") " pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:29.727999 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.727980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czp5m\" (UniqueName: \"kubernetes.io/projected/13702528-9a6f-4a00-8871-053cbe8163e7-kube-api-access-czp5m\") pod \"test-curl-10810-1c0595\" (UID: \"13702528-9a6f-4a00-8871-053cbe8163e7\") " pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:29.810627 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:29.810580 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:30.078368 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:30.078291 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10810-1c0595"] Sep 17 01:55:30.081846 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:55:30.081820 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13702528_9a6f_4a00_8871_053cbe8163e7.slice/crio-a3a4fa354e0427c47c13b598e7ce6d5c5fd00949b5d43532a6ec794a239a3a75 WatchSource:0}: Error finding container a3a4fa354e0427c47c13b598e7ce6d5c5fd00949b5d43532a6ec794a239a3a75: Status 404 returned error can't find the container with id a3a4fa354e0427c47c13b598e7ce6d5c5fd00949b5d43532a6ec794a239a3a75 Sep 17 01:55:30.218582 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:30.218555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10810-1c0595" event={"ID":"13702528-9a6f-4a00-8871-053cbe8163e7","Type":"ContainerStarted","Data":"a3a4fa354e0427c47c13b598e7ce6d5c5fd00949b5d43532a6ec794a239a3a75"} Sep 17 01:55:31.089039 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:31.088982 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:31.179507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:31.179468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:32.338703 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:32.338671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:32.438319 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:32.438285 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:32.536616 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:32.536590 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:32.952361 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:32.952321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:33.059269 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.059234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:33.235739 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.235709 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10810-1c0595" event={"ID":"13702528-9a6f-4a00-8871-053cbe8163e7","Type":"ContainerStarted","Data":"ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73"} Sep 17 01:55:33.250836 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.250793 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10810-1c0595" podStartSLOduration=1.9718875009999999 podStartE2EDuration="4.250778001s" totalImagesPullingTime="2.2788905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:33.250079922 +0000 UTC m=+1078.085132259" watchObservedRunningTime="2026-09-17 01:55:33.250778001 +0000 UTC m=+1078.085830328" Sep 17 01:55:33.553169 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.553087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:33.696480 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.696443 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:33.783238 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.783196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:33.877241 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.877152 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10810-1c0595"] Sep 17 01:55:33.893249 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.893225 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10810-1c0595"] Sep 17 01:55:33.896675 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.896633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:33.990047 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:33.990021 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:34.024079 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.024050 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10810-940810"] Sep 17 01:55:34.024403 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.024387 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" containerName="test-curl-10810-1c0595" Sep 17 01:55:34.024403 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.024402 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" containerName="test-curl-10810-1c0595" Sep 17 01:55:34.024543 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.024485 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" containerName="test-curl-10810-1c0595" Sep 17 01:55:34.027446 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.027427 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:34.032855 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.032761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10810-940810"] Sep 17 01:55:34.156240 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.156166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wln8\" (UniqueName: \"kubernetes.io/projected/a25eb2e5-2346-445d-8886-d36c2e785873-kube-api-access-2wln8\") pod \"test-curl-10810-940810\" (UID: \"a25eb2e5-2346-445d-8886-d36c2e785873\") " pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:34.241481 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:34.241452 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10810-1c0595" container="test-curl-10810-1c0595" Sep 17 01:55:34.256889 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.256867 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wln8\" (UniqueName: \"kubernetes.io/projected/a25eb2e5-2346-445d-8886-d36c2e785873-kube-api-access-2wln8\") pod \"test-curl-10810-940810\" (UID: \"a25eb2e5-2346-445d-8886-d36c2e785873\") " pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:34.264673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.264640 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wln8\" (UniqueName: \"kubernetes.io/projected/a25eb2e5-2346-445d-8886-d36c2e785873-kube-api-access-2wln8\") pod \"test-curl-10810-940810\" (UID: \"a25eb2e5-2346-445d-8886-d36c2e785873\") " pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:34.338859 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.338837 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:34.606437 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.606411 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10810-940810"] Sep 17 01:55:34.608633 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:55:34.608608 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25eb2e5_2346_445d_8886_d36c2e785873.slice/crio-022f69ae6b060a467628fc13ec5e00f1ff960c3b358a9a74c9df10c828359c3e WatchSource:0}: Error finding container 022f69ae6b060a467628fc13ec5e00f1ff960c3b358a9a74c9df10c828359c3e: Status 404 returned error can't find the container with id 022f69ae6b060a467628fc13ec5e00f1ff960c3b358a9a74c9df10c828359c3e Sep 17 01:55:34.611609 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:34.611580 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:35.247940 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.247900 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10810-940810" event={"ID":"a25eb2e5-2346-445d-8886-d36c2e785873","Type":"ContainerStarted","Data":"484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60"} Sep 17 01:55:35.247940 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.247948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10810-940810" event={"ID":"a25eb2e5-2346-445d-8886-d36c2e785873","Type":"ContainerStarted","Data":"022f69ae6b060a467628fc13ec5e00f1ff960c3b358a9a74c9df10c828359c3e"} Sep 17 01:55:35.248180 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.248030 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10810-1c0595" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" containerName="test-curl-10810-1c0595" containerID="cri-o://ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73" gracePeriod=2 Sep 17 01:55:35.262289 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.262245 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10810-940810" podStartSLOduration=1.2622051 podStartE2EDuration="1.2622051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:35.261295562 +0000 UTC m=+1080.096347898" watchObservedRunningTime="2026-09-17 01:55:35.2622051 +0000 UTC m=+1080.097257433" Sep 17 01:55:35.263091 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:35.263068 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10810-1c0595\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:35.279336 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.279310 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:55:35.616527 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.616446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:35.676987 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:35.676947 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10810-1c0595\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:35.718625 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.718592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:35.819810 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.819760 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:35.929907 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.929844 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10810-940810"] Sep 17 01:55:35.933522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:35.933499 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10810-940810"] Sep 17 01:55:36.101265 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.101234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:36.198975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.198888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:36.238997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.238972 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp"] Sep 17 01:55:36.239367 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.239354 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" containerName="test-curl-10810-940810" Sep 17 01:55:36.239427 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.239369 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" containerName="test-curl-10810-940810" Sep 17 01:55:36.239427 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.239425 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" containerName="test-curl-10810-940810" Sep 17 01:55:36.242404 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.242389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.244884 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.244866 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e564e1\"" Sep 17 01:55:36.244969 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.244872 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e564e1-dockercfg-xsrdn\"" Sep 17 01:55:36.252154 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.252128 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp"] Sep 17 01:55:36.253514 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:36.253497 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10810-940810" container="test-curl-10810-940810" Sep 17 01:55:36.375447 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.375418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn2g\" (UniqueName: \"kubernetes.io/projected/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-kube-api-access-ljn2g\") pod \"payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.375588 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.375484 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.476664 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.476641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn2g\" (UniqueName: \"kubernetes.io/projected/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-kube-api-access-ljn2g\") pod \"payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.476782 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.476677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.477321 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.477305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.484297 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.484277 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn2g\" (UniqueName: \"kubernetes.io/projected/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-kube-api-access-ljn2g\") pod \"payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.553505 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.553483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:36.558609 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.558586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:36.660511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.660482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:36.756298 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.756218 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:36.831023 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.831001 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp"] Sep 17 01:55:36.833298 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:55:36.833273 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3d2bb3_ceed_4c65_8203_e4008b9ac0c2.slice/crio-a6fc910b94b1196008bd8a5ae18ce08d6bf8dc8c3f5ca09a5589f855705a7f8b WatchSource:0}: Error finding container a6fc910b94b1196008bd8a5ae18ce08d6bf8dc8c3f5ca09a5589f855705a7f8b: Status 404 returned error can't find the container with id a6fc910b94b1196008bd8a5ae18ce08d6bf8dc8c3f5ca09a5589f855705a7f8b Sep 17 01:55:36.900740 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:36.900712 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:37.160099 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.160011 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:37.209880 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.209851 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6"] Sep 17 01:55:37.214833 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.214810 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.217039 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.217015 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-266b61f5-openshift-default-dockercfg-s52ss\"" Sep 17 01:55:37.217379 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.217357 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 01:55:37.228449 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.228426 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6"] Sep 17 01:55:37.256001 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.255753 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:37.269131 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.269101 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" event={"ID":"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2","Type":"ContainerStarted","Data":"a6fc910b94b1196008bd8a5ae18ce08d6bf8dc8c3f5ca09a5589f855705a7f8b"} Sep 17 01:55:37.269304 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.269276 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10810-940810" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" containerName="test-curl-10810-940810" containerID="cri-o://484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60" gracePeriod=2 Sep 17 01:55:37.272084 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:37.272058 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10810-940810\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:37.336389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.336320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:37.384883 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.384861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-data\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.384987 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.384911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-socket\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.384987 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.384934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-certs\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.384987 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.384971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istiod-ca-cert\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.385117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.385011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-token\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.385117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.385041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-envoy\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.385117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.385085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srqw\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-kube-api-access-8srqw\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.385117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.385115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-podinfo\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.385283 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.385146 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-credential-socket\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.390647 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.390626 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:37.392727 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:37.392700 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10810-940810\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:37.486134 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486106 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czp5m\" (UniqueName: \"kubernetes.io/projected/13702528-9a6f-4a00-8871-053cbe8163e7-kube-api-access-czp5m\") pod \"13702528-9a6f-4a00-8871-053cbe8163e7\" (UID: \"13702528-9a6f-4a00-8871-053cbe8163e7\") " Sep 17 01:55:37.486559 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istiod-ca-cert\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486714 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-token\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486714 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-envoy\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486714 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srqw\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-kube-api-access-8srqw\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486712 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-podinfo\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-credential-socket\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-data\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486877 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-socket\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.486891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.486912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-certs\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.487973 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.487261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-certs\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.487973 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.487289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istiod-ca-cert\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.487973 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.487515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-credential-socket\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.488237 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.487997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-data\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.488237 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.488186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-socket\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.490566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.490528 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-envoy\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.491584 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.491561 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13702528-9a6f-4a00-8871-053cbe8163e7-kube-api-access-czp5m" (OuterVolumeSpecName: "kube-api-access-czp5m") pod "13702528-9a6f-4a00-8871-053cbe8163e7" (UID: "13702528-9a6f-4a00-8871-053cbe8163e7"). InnerVolumeSpecName "kube-api-access-czp5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:55:37.492753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.492733 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-podinfo\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.497229 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.497172 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-token\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.497764 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.497724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srqw\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-kube-api-access-8srqw\") pod \"e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.528078 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.527902 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:37.588329 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.588295 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czp5m\" (UniqueName: \"kubernetes.io/projected/13702528-9a6f-4a00-8871-053cbe8163e7-kube-api-access-czp5m\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:55:37.657754 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.657717 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:37.682282 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.682253 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13702528-9a6f-4a00-8871-053cbe8163e7" path="/var/lib/kubelet/pods/13702528-9a6f-4a00-8871-053cbe8163e7/volumes" Sep 17 01:55:37.759120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.759011 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:37.822812 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:37.822714 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6"] Sep 17 01:55:37.826680 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:55:37.826651 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b17fa37_01a6_48ef_8ee7_1f3039e0bce4.slice/crio-540831c23386bf836592bfe96dd9b7ce22acc78633ab4e90b564194d00114dcd WatchSource:0}: Error finding container 540831c23386bf836592bfe96dd9b7ce22acc78633ab4e90b564194d00114dcd: Status 404 returned error can't find the container with id 540831c23386bf836592bfe96dd9b7ce22acc78633ab4e90b564194d00114dcd Sep 17 01:55:38.275311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:38.275277 2582 generic.go:356] "Generic (PLEG): container finished" podID="13702528-9a6f-4a00-8871-053cbe8163e7" containerID="ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73" exitCode=137 Sep 17 01:55:38.275497 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:38.275346 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-1c0595" Sep 17 01:55:38.275497 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:38.275364 2582 scope.go:117] "RemoveContainer" containerID="ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73" Sep 17 01:55:38.277261 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:38.277186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" event={"ID":"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4","Type":"ContainerStarted","Data":"540831c23386bf836592bfe96dd9b7ce22acc78633ab4e90b564194d00114dcd"} Sep 17 01:55:38.301955 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:38.301909 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:39.162521 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.162360 2582 scope.go:117] "RemoveContainer" containerID="ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73" Sep 17 01:55:39.162734 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:39.162688 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73\": container with ID starting with ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73 not found: ID does not exist" containerID="ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73" Sep 17 01:55:39.162776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.162726 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73"} err="failed to get container status \"ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73\": rpc error: code = NotFound desc = could not find container \"ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73\": container with ID starting with ac012ea5cee921eb0abfb1e0a508680b4a3f9bd9fe34dac96fd24671dd9d2f73 not found: ID does not exist" Sep 17 01:55:39.206063 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.206036 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:39.289757 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.289716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" event={"ID":"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2","Type":"ContainerStarted","Data":"413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179"} Sep 17 01:55:39.289933 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.289813 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:39.308268 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.308176 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" podStartSLOduration=0.931539362 podStartE2EDuration="3.308156511s" totalImagesPullingTime="2.376617149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:39.305995497 +0000 UTC m=+1084.141047832" watchObservedRunningTime="2026-09-17 01:55:39.308156511 +0000 UTC m=+1084.143208838" Sep 17 01:55:39.435556 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.435532 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:39.603937 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.603905 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wln8\" (UniqueName: \"kubernetes.io/projected/a25eb2e5-2346-445d-8886-d36c2e785873-kube-api-access-2wln8\") pod \"a25eb2e5-2346-445d-8886-d36c2e785873\" (UID: \"a25eb2e5-2346-445d-8886-d36c2e785873\") " Sep 17 01:55:39.605878 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.605848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25eb2e5-2346-445d-8886-d36c2e785873-kube-api-access-2wln8" (OuterVolumeSpecName: "kube-api-access-2wln8") pod "a25eb2e5-2346-445d-8886-d36c2e785873" (UID: "a25eb2e5-2346-445d-8886-d36c2e785873"). InnerVolumeSpecName "kube-api-access-2wln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:55:39.680476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.680412 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25eb2e5-2346-445d-8886-d36c2e785873" path="/var/lib/kubelet/pods/a25eb2e5-2346-445d-8886-d36c2e785873/volumes" Sep 17 01:55:39.705405 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:39.705381 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wln8\" (UniqueName: \"kubernetes.io/projected/a25eb2e5-2346-445d-8886-d36c2e785873-kube-api-access-2wln8\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:55:40.159483 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:40.159449 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:40.297292 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:40.297259 2582 generic.go:356] "Generic (PLEG): container finished" podID="a25eb2e5-2346-445d-8886-d36c2e785873" containerID="484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60" exitCode=137 Sep 17 01:55:40.297741 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:40.297309 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10810-940810" Sep 17 01:55:40.297741 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:40.297371 2582 scope.go:117] "RemoveContainer" containerID="484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60" Sep 17 01:55:41.210712 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:41.210692 2582 scope.go:117] "RemoveContainer" containerID="484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60" Sep 17 01:55:41.211061 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:55:41.211040 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60\": container with ID starting with 484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60 not found: ID does not exist" containerID="484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60" Sep 17 01:55:41.211124 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:41.211071 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60"} err="failed to get container status \"484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60\": rpc error: code = NotFound desc = could not find container \"484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60\": container with ID starting with 484916a64fcb5ba3a70b5e80d7bec9222cfa9b43fa3011b9ba45df33276a1e60 not found: ID does not exist" Sep 17 01:55:41.270896 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:41.270856 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:55:41.271005 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:41.270949 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:55:41.271005 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:41.270995 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:55:42.309683 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:42.309649 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" event={"ID":"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4","Type":"ContainerStarted","Data":"67e80b6a35e0e515a36202a81c9fb53930fbd87c40253b4fc58b3895eac76fca"} Sep 17 01:55:42.360012 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:42.359970 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" podStartSLOduration=1.9183608909999998 podStartE2EDuration="5.359956168s" totalImagesPullingTime="3.441595277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:42.359127273 +0000 UTC m=+1087.194179607" watchObservedRunningTime="2026-09-17 01:55:42.359956168 +0000 UTC m=+1087.195008501" Sep 17 01:55:42.528978 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:42.528943 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:42.533853 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:42.533828 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:43.281157 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:43.281123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:43.314984 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:43.314964 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:43.315865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:43.315845 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:55:43.794764 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:43.794726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:43.891331 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:43.891280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:44.277076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:44.277038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:45.868004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:45.867974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:47.560605 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:47.560559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:47.846430 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:47.846355 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:47.986146 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:47.986120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:48.152119 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:48.152050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:48.482430 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:48.482405 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:55:48.512649 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:48.512604 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:48.999725 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:48.999693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:49.180454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:49.180415 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:49.353296 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:49.353227 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:50.097492 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.097455 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:50.299153 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.299124 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 01:55:50.384022 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.383939 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:50.431258 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.431201 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7"] Sep 17 01:55:50.435632 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.435614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.443076 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.442868 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-266b61f5\"" Sep 17 01:55:50.443226 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.443194 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-266b61f5-dockercfg-lpqws\"" Sep 17 01:55:50.467410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.467387 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7"] Sep 17 01:55:50.496046 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.496025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a36450b-91aa-49b6-bf8b-6d63a8eee734-plugins-config-volume\") pod \"payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.496169 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.496081 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmfq\" (UniqueName: \"kubernetes.io/projected/4a36450b-91aa-49b6-bf8b-6d63a8eee734-kube-api-access-khmfq\") pod \"payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.596792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.596769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmfq\" (UniqueName: \"kubernetes.io/projected/4a36450b-91aa-49b6-bf8b-6d63a8eee734-kube-api-access-khmfq\") pod \"payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.596909 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.596858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a36450b-91aa-49b6-bf8b-6d63a8eee734-plugins-config-volume\") pod \"payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.597530 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.597512 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a36450b-91aa-49b6-bf8b-6d63a8eee734-plugins-config-volume\") pod \"payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.608907 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.608888 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmfq\" (UniqueName: \"kubernetes.io/projected/4a36450b-91aa-49b6-bf8b-6d63a8eee734-kube-api-access-khmfq\") pod \"payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:50.746626 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:50.746603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:51.017647 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.017552 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:51.025645 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.025594 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7"] Sep 17 01:55:51.028327 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:55:51.028302 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a36450b_91aa_49b6_bf8b_6d63a8eee734.slice/crio-08501e8146a27766f711adb5b2915a9d933ce857b27405a2a170830cc8e282db WatchSource:0}: Error finding container 08501e8146a27766f711adb5b2915a9d933ce857b27405a2a170830cc8e282db: Status 404 returned error can't find the container with id 08501e8146a27766f711adb5b2915a9d933ce857b27405a2a170830cc8e282db Sep 17 01:55:51.122332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.122290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:51.353387 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.353309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" event={"ID":"4a36450b-91aa-49b6-bf8b-6d63a8eee734","Type":"ContainerStarted","Data":"df7434312e7dcae65e32647f54cd91c0947bea4cd5da89668589ff313a51e334"} Sep 17 01:55:51.353387 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.353366 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" event={"ID":"4a36450b-91aa-49b6-bf8b-6d63a8eee734","Type":"ContainerStarted","Data":"08501e8146a27766f711adb5b2915a9d933ce857b27405a2a170830cc8e282db"} Sep 17 01:55:51.353585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.353417 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:55:51.362940 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.362911 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:51.378143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.378106 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" podStartSLOduration=1.378091787 podStartE2EDuration="1.378091787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:51.377455227 +0000 UTC m=+1096.212507563" watchObservedRunningTime="2026-09-17 01:55:51.378091787 +0000 UTC m=+1096.213144120" Sep 17 01:55:51.872840 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:51.872808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:52.171476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:52.171392 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:52.331304 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:52.331274 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:53.373084 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:53.373052 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:53.717979 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:53.717945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:54.193971 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:54.193866 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:54.324762 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:54.324708 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:54.971410 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:54.971374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:55.562412 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:55.562371 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:55.655041 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:55.655002 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:56.492343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:56.492309 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:56.864733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:56.864642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:57.063745 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:57.063705 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:57.245401 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:57.245368 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:57.404337 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:57.404303 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:58.400561 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:58.400525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:59.164121 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.164088 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c"] Sep 17 01:55:59.172647 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.172625 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:55:59.175627 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.175603 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 01:55:59.175775 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.175603 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:55:59.176782 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.176758 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c"] Sep 17 01:55:59.281092 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.281066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b47864-bdc0-4160-b137-b1e48804e71f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c\" (UID: \"e7b47864-bdc0-4160-b137-b1e48804e71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:55:59.382241 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.382194 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b47864-bdc0-4160-b137-b1e48804e71f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c\" (UID: \"e7b47864-bdc0-4160-b137-b1e48804e71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:55:59.382778 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.382760 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b47864-bdc0-4160-b137-b1e48804e71f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c\" (UID: \"e7b47864-bdc0-4160-b137-b1e48804e71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:55:59.485314 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.485288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:55:59.690928 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.690868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:55:59.789409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.789382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c"] Sep 17 01:55:59.790773 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:55:59.790739 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b47864_bdc0_4160_b137_b1e48804e71f.slice/crio-a11b60846f35334deb1f7841205007729c14a00f9e7913e173c237235b53c183 WatchSource:0}: Error finding container a11b60846f35334deb1f7841205007729c14a00f9e7913e173c237235b53c183: Status 404 returned error can't find the container with id a11b60846f35334deb1f7841205007729c14a00f9e7913e173c237235b53c183 Sep 17 01:55:59.797906 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:55:59.797887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:00.157454 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.157365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:00.397327 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.397282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" event={"ID":"e7b47864-bdc0-4160-b137-b1e48804e71f","Type":"ContainerStarted","Data":"a11b60846f35334deb1f7841205007729c14a00f9e7913e173c237235b53c183"} Sep 17 01:56:00.538233 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.537906 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:00.640236 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.637118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:00.670701 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.670666 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w"] Sep 17 01:56:00.674464 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.674440 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.679606 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.679467 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55c648\"" Sep 17 01:56:00.679606 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.679503 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55c648-dockercfg-94kh6\"" Sep 17 01:56:00.691782 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.691700 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w"] Sep 17 01:56:00.796135 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.796012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m475z\" (UniqueName: \"kubernetes.io/projected/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-kube-api-access-m475z\") pod \"payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.796334 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.796148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.897137 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.897099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m475z\" (UniqueName: \"kubernetes.io/projected/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-kube-api-access-m475z\") pod \"payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.897338 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.897148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.898079 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.898053 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.909021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.908994 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m475z\" (UniqueName: \"kubernetes.io/projected/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-kube-api-access-m475z\") pod \"payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:00.950677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.950624 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:00.989660 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:00.989622 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:01.286323 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:01.286290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w"] Sep 17 01:56:01.289308 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:01.289274 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0e3b35_b7bc_4db8_834e_3b88f16f7665.slice/crio-cfd30c68656b25a49bacb0a4abc353a45b2d7d3b2541f8d158ea4ad9b2ebe6ed WatchSource:0}: Error finding container cfd30c68656b25a49bacb0a4abc353a45b2d7d3b2541f8d158ea4ad9b2ebe6ed: Status 404 returned error can't find the container with id cfd30c68656b25a49bacb0a4abc353a45b2d7d3b2541f8d158ea4ad9b2ebe6ed Sep 17 01:56:01.406912 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:01.406692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" event={"ID":"ea0e3b35-b7bc-4db8-834e-3b88f16f7665","Type":"ContainerStarted","Data":"0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4"} Sep 17 01:56:01.406912 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:01.406754 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" event={"ID":"ea0e3b35-b7bc-4db8-834e-3b88f16f7665","Type":"ContainerStarted","Data":"cfd30c68656b25a49bacb0a4abc353a45b2d7d3b2541f8d158ea4ad9b2ebe6ed"} Sep 17 01:56:01.406912 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:01.406796 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:01.454748 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:01.454688 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" podStartSLOduration=1.454670615 podStartE2EDuration="1.454670615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:01.454267155 +0000 UTC m=+1106.289319489" watchObservedRunningTime="2026-09-17 01:56:01.454670615 +0000 UTC m=+1106.289722982" Sep 17 01:56:01.993551 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:01.993510 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:02.361128 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:02.361051 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:56:02.909226 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:02.909163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:03.181295 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.181260 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7"] Sep 17 01:56:03.184778 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.184755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.191020 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.190984 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"kube-root-ca.crt\"" Sep 17 01:56:03.191194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.191020 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"openshift-service-ca.crt\"" Sep 17 01:56:03.191194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.190996 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"simulator-w0-e564e1-kserve-self-signed-certs\"" Sep 17 01:56:03.191313 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.191049 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"default-dockercfg-2ftzh\"" Sep 17 01:56:03.202108 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.202072 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7"] Sep 17 01:56:03.287191 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.287149 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:03.338104 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.338018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-dshm\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.338104 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.338093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tls-certs\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.338646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.338192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-home\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.338646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.338243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-model-cache\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.338646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.338299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tmp-dir\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.338646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.338449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssrr\" (UniqueName: \"kubernetes.io/projected/63d24bc1-2dab-44fd-b5d7-d529d29a443b-kube-api-access-4ssrr\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.421426 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.421319 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" event={"ID":"e7b47864-bdc0-4160-b137-b1e48804e71f","Type":"ContainerStarted","Data":"b481eb22a224ff66d2d915fe55c4523d00cea2121d4461e8895876e2990a4a9e"} Sep 17 01:56:03.439177 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssrr\" (UniqueName: \"kubernetes.io/projected/63d24bc1-2dab-44fd-b5d7-d529d29a443b-kube-api-access-4ssrr\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.439376 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-dshm\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.439376 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tls-certs\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.439540 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-home\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.439680 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-model-cache\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.439680 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tmp-dir\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.439911 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439885 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-home\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.440013 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.439938 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tmp-dir\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.440119 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.440100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-model-cache\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.441729 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.441669 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" podStartSLOduration=1.544623769 podStartE2EDuration="4.441650887s" totalImagesPullingTime="2.897027118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:03.438066037 +0000 UTC m=+1108.273118370" watchObservedRunningTime="2026-09-17 01:56:03.441650887 +0000 UTC m=+1108.276703219" Sep 17 01:56:03.442063 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.442039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-dshm\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.442419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.442397 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tls-certs\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.448853 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.448830 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssrr\" (UniqueName: \"kubernetes.io/projected/63d24bc1-2dab-44fd-b5d7-d529d29a443b-kube-api-access-4ssrr\") pod \"simulator-w0-e564e1-kserve-94d859b59-tckb7\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.503270 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.503203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:03.803804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.803776 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7"] Sep 17 01:56:03.806057 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:03.806021 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63d24bc1_2dab_44fd_b5d7_d529d29a443b.slice/crio-18cb34f771a717d6c68f5ee971aced6f99a45844b20d727258e0a287e6e4d76b WatchSource:0}: Error finding container 18cb34f771a717d6c68f5ee971aced6f99a45844b20d727258e0a287e6e4d76b: Status 404 returned error can't find the container with id 18cb34f771a717d6c68f5ee971aced6f99a45844b20d727258e0a287e6e4d76b Sep 17 01:56:03.837644 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:03.837600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:04.271102 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.271064 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:04.378718 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.378675 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:04.428723 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.428684 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" event={"ID":"63d24bc1-2dab-44fd-b5d7-d529d29a443b","Type":"ContainerStarted","Data":"aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769"} Sep 17 01:56:04.428925 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.428735 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" event={"ID":"63d24bc1-2dab-44fd-b5d7-d529d29a443b","Type":"ContainerStarted","Data":"18cb34f771a717d6c68f5ee971aced6f99a45844b20d727258e0a287e6e4d76b"} Sep 17 01:56:04.430563 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.430519 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7b47864-bdc0-4160-b137-b1e48804e71f" containerID="b481eb22a224ff66d2d915fe55c4523d00cea2121d4461e8895876e2990a4a9e" exitCode=0 Sep 17 01:56:04.430724 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.430586 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" event={"ID":"e7b47864-bdc0-4160-b137-b1e48804e71f","Type":"ContainerDied","Data":"b481eb22a224ff66d2d915fe55c4523d00cea2121d4461e8895876e2990a4a9e"} Sep 17 01:56:04.462929 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.462867 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" podStartSLOduration=1.462846285 podStartE2EDuration="1.462846285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:04.461908049 +0000 UTC m=+1109.296960382" watchObservedRunningTime="2026-09-17 01:56:04.462846285 +0000 UTC m=+1109.297898618" Sep 17 01:56:04.777980 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:04.777925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:05.573344 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:05.573274 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:56:05.611389 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:05.611359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:05.661946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:05.661909 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b47864-bdc0-4160-b137-b1e48804e71f-maas-api-service-ca\") pod \"e7b47864-bdc0-4160-b137-b1e48804e71f\" (UID: \"e7b47864-bdc0-4160-b137-b1e48804e71f\") " Sep 17 01:56:05.662325 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:05.662299 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b47864-bdc0-4160-b137-b1e48804e71f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7b47864-bdc0-4160-b137-b1e48804e71f" (UID: "e7b47864-bdc0-4160-b137-b1e48804e71f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:05.762585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:05.762547 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b47864-bdc0-4160-b137-b1e48804e71f-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:06.441408 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.441372 2582 generic.go:356] "Generic (PLEG): container finished" podID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerID="67e80b6a35e0e515a36202a81c9fb53930fbd87c40253b4fc58b3895eac76fca" exitCode=0 Sep 17 01:56:06.441660 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.441443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" event={"ID":"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4","Type":"ContainerDied","Data":"67e80b6a35e0e515a36202a81c9fb53930fbd87c40253b4fc58b3895eac76fca"} Sep 17 01:56:06.441911 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.441892 2582 scope.go:117] "RemoveContainer" containerID="67e80b6a35e0e515a36202a81c9fb53930fbd87c40253b4fc58b3895eac76fca" Sep 17 01:56:06.442997 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.442978 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" Sep 17 01:56:06.443089 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.443030 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:56:06.443089 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.442975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c" event={"ID":"e7b47864-bdc0-4160-b137-b1e48804e71f","Type":"ContainerDied","Data":"a11b60846f35334deb1f7841205007729c14a00f9e7913e173c237235b53c183"} Sep 17 01:56:06.443192 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.443101 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a11b60846f35334deb1f7841205007729c14a00f9e7913e173c237235b53c183" Sep 17 01:56:06.443192 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.443102 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:56:06.443311 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:06.443191 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:56:07.452911 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:07.452876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" event={"ID":"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4","Type":"ContainerStarted","Data":"55e000994b1844ac75a0ba47a5f9132418962573e786f73aa3fe596dc35f0611"} Sep 17 01:56:07.531243 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:07.530440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:56:07.531243 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:07.530690 2582 patch_prober.go:28] interesting pod/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 17 01:56:07.531243 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:07.530749 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 17 01:56:07.900444 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:07.900405 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:56:08.529462 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:08.529418 2582 patch_prober.go:28] interesting pod/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 17 01:56:08.529965 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:08.529494 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 17 01:56:09.276402 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.276371 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66"] Sep 17 01:56:09.276802 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.276785 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b47864-bdc0-4160-b137-b1e48804e71f" containerName="revoke-keys" Sep 17 01:56:09.276870 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.276804 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b47864-bdc0-4160-b137-b1e48804e71f" containerName="revoke-keys" Sep 17 01:56:09.276908 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.276899 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b47864-bdc0-4160-b137-b1e48804e71f" containerName="revoke-keys" Sep 17 01:56:09.279114 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.279098 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.281797 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.281760 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3f027f\"/\"default-dockercfg-ztf9z\"" Sep 17 01:56:09.281797 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.281792 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3f027f\"/\"kube-root-ca.crt\"" Sep 17 01:56:09.282532 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.282512 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3f027f\"/\"openshift-service-ca.crt\"" Sep 17 01:56:09.282811 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.282783 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3f027f\"/\"simulator-w3-3f027f-kserve-self-signed-certs\"" Sep 17 01:56:09.291328 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.291307 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66"] Sep 17 01:56:09.301934 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.301904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:09.394848 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.394813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-home\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.395021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.394859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-dshm\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.395021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.394926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-tmp-dir\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.395021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.394944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-model-cache\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.395021 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.395005 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9a3bea-202b-475e-afc6-42c7deb6b192-tls-certs\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.395171 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.395035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfld\" (UniqueName: \"kubernetes.io/projected/ca9a3bea-202b-475e-afc6-42c7deb6b192-kube-api-access-smfld\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.407122 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.407081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:09.496285 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-home\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496285 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-dshm\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-tmp-dir\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-model-cache\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9a3bea-202b-475e-afc6-42c7deb6b192-tls-certs\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfld\" (UniqueName: \"kubernetes.io/projected/ca9a3bea-202b-475e-afc6-42c7deb6b192-kube-api-access-smfld\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496773 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-home\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496851 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-model-cache\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.496964 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.496927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-tmp-dir\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.498689 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.498667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-dshm\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.499098 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.499078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9a3bea-202b-475e-afc6-42c7deb6b192-tls-certs\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.504295 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.504274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfld\" (UniqueName: \"kubernetes.io/projected/ca9a3bea-202b-475e-afc6-42c7deb6b192-kube-api-access-smfld\") pod \"simulator-w3-3f027f-kserve-5fff47f766-bnr66\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.529602 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.529494 2582 patch_prober.go:28] interesting pod/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 17 01:56:09.529602 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.529573 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 17 01:56:09.588396 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.588341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:09.889522 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:09.889496 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66"] Sep 17 01:56:09.891672 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:09.891645 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca9a3bea_202b_475e_afc6_42c7deb6b192.slice/crio-ff639236d047f90dc3a60579344be4bcd219263fdf1efb7671e27ef21e5a8f13 WatchSource:0}: Error finding container ff639236d047f90dc3a60579344be4bcd219263fdf1efb7671e27ef21e5a8f13: Status 404 returned error can't find the container with id ff639236d047f90dc3a60579344be4bcd219263fdf1efb7671e27ef21e5a8f13 Sep 17 01:56:10.237854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.237775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:10.262769 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.262725 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7"] Sep 17 01:56:10.263014 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.262983 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" podUID="4a36450b-91aa-49b6-bf8b-6d63a8eee734" containerName="payload-processing" containerID="cri-o://df7434312e7dcae65e32647f54cd91c0947bea4cd5da89668589ff313a51e334" gracePeriod=30 Sep 17 01:56:10.329968 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.329932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:10.430924 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.430841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:10.450548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.450515 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c"] Sep 17 01:56:10.455555 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.455523 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-266b61f5-fjv7c"] Sep 17 01:56:10.471564 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.471538 2582 generic.go:356] "Generic (PLEG): container finished" podID="4a36450b-91aa-49b6-bf8b-6d63a8eee734" containerID="df7434312e7dcae65e32647f54cd91c0947bea4cd5da89668589ff313a51e334" exitCode=0 Sep 17 01:56:10.471790 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.471584 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" event={"ID":"4a36450b-91aa-49b6-bf8b-6d63a8eee734","Type":"ContainerDied","Data":"df7434312e7dcae65e32647f54cd91c0947bea4cd5da89668589ff313a51e334"} Sep 17 01:56:10.474344 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.474325 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" event={"ID":"ca9a3bea-202b-475e-afc6-42c7deb6b192","Type":"ContainerStarted","Data":"d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf"} Sep 17 01:56:10.474483 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.474465 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" event={"ID":"ca9a3bea-202b-475e-afc6-42c7deb6b192","Type":"ContainerStarted","Data":"ff639236d047f90dc3a60579344be4bcd219263fdf1efb7671e27ef21e5a8f13"} Sep 17 01:56:10.491767 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.491658 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" podStartSLOduration=1.491641156 podStartE2EDuration="1.491641156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:10.490485334 +0000 UTC m=+1115.325537668" watchObservedRunningTime="2026-09-17 01:56:10.491641156 +0000 UTC m=+1115.326693489" Sep 17 01:56:10.539030 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.538989 2582 patch_prober.go:28] interesting pod/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 17 01:56:10.539975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.539884 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 17 01:56:10.547062 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.546557 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:10.575761 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.574928 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:56:10.596263 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.596199 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6"] Sep 17 01:56:10.596511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.596485 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" containerID="cri-o://55e000994b1844ac75a0ba47a5f9132418962573e786f73aa3fe596dc35f0611" gracePeriod=30 Sep 17 01:56:10.644827 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.644763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:10.709343 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.709312 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khmfq\" (UniqueName: \"kubernetes.io/projected/4a36450b-91aa-49b6-bf8b-6d63a8eee734-kube-api-access-khmfq\") pod \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " Sep 17 01:56:10.709513 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.709421 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a36450b-91aa-49b6-bf8b-6d63a8eee734-plugins-config-volume\") pod \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\" (UID: \"4a36450b-91aa-49b6-bf8b-6d63a8eee734\") " Sep 17 01:56:10.709920 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.709892 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a36450b-91aa-49b6-bf8b-6d63a8eee734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a36450b-91aa-49b6-bf8b-6d63a8eee734" (UID: "4a36450b-91aa-49b6-bf8b-6d63a8eee734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:10.724236 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.720746 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a36450b-91aa-49b6-bf8b-6d63a8eee734-kube-api-access-khmfq" (OuterVolumeSpecName: "kube-api-access-khmfq") pod "4a36450b-91aa-49b6-bf8b-6d63a8eee734" (UID: "4a36450b-91aa-49b6-bf8b-6d63a8eee734"). InnerVolumeSpecName "kube-api-access-khmfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:10.810566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.810471 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmfq\" (UniqueName: \"kubernetes.io/projected/4a36450b-91aa-49b6-bf8b-6d63a8eee734-kube-api-access-khmfq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:10.810566 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.810507 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a36450b-91aa-49b6-bf8b-6d63a8eee734-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:10.869985 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:10.869952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:11.200174 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.200090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:11.292601 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.292565 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:11.434023 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.433990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:11.480379 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.480356 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" Sep 17 01:56:11.480513 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.480399 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7" event={"ID":"4a36450b-91aa-49b6-bf8b-6d63a8eee734","Type":"ContainerDied","Data":"08501e8146a27766f711adb5b2915a9d933ce857b27405a2a170830cc8e282db"} Sep 17 01:56:11.480513 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.480460 2582 scope.go:117] "RemoveContainer" containerID="df7434312e7dcae65e32647f54cd91c0947bea4cd5da89668589ff313a51e334" Sep 17 01:56:11.506040 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.506019 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7"] Sep 17 01:56:11.509556 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.509535 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-266b61f5-7664d69bc-hdcv7"] Sep 17 01:56:11.680118 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.680083 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a36450b-91aa-49b6-bf8b-6d63a8eee734" path="/var/lib/kubelet/pods/4a36450b-91aa-49b6-bf8b-6d63a8eee734/volumes" Sep 17 01:56:11.680499 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.680444 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b47864-bdc0-4160-b137-b1e48804e71f" path="/var/lib/kubelet/pods/e7b47864-bdc0-4160-b137-b1e48804e71f/volumes" Sep 17 01:56:11.757264 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.757159 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:11.853877 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:11.853841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:12.417518 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:12.417476 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 01:56:12.666446 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:12.666400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:13.315137 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:13.315098 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:56:13.504192 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:13.504151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:13.504422 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:13.504249 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:13.517164 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:13.517130 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:13.592801 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:13.592698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:14.510570 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:14.510539 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 01:56:14.723876 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:14.723839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:14.906873 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:14.906785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:15.398942 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:15.398907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:15.984043 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:15.984014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:17.035754 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:17.035719 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:17.682649 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:17.682610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:17.783830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:17.783790 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:18.008511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.008478 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:18.342839 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.342768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:18.436722 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.436691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:18.483549 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.483522 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p"] Sep 17 01:56:18.483975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.483957 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a36450b-91aa-49b6-bf8b-6d63a8eee734" containerName="payload-processing" Sep 17 01:56:18.484057 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.483977 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a36450b-91aa-49b6-bf8b-6d63a8eee734" containerName="payload-processing" Sep 17 01:56:18.484117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.484098 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a36450b-91aa-49b6-bf8b-6d63a8eee734" containerName="payload-processing" Sep 17 01:56:18.486051 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.486030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.488475 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.488455 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-404126c3-dockercfg-h7cfp\"" Sep 17 01:56:18.488576 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.488511 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-404126c3\"" Sep 17 01:56:18.498651 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.498630 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p"] Sep 17 01:56:18.520611 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.520583 2582 generic.go:356] "Generic (PLEG): container finished" podID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerID="55e000994b1844ac75a0ba47a5f9132418962573e786f73aa3fe596dc35f0611" exitCode=0 Sep 17 01:56:18.520699 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.520651 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" event={"ID":"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4","Type":"ContainerDied","Data":"55e000994b1844ac75a0ba47a5f9132418962573e786f73aa3fe596dc35f0611"} Sep 17 01:56:18.520745 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.520701 2582 scope.go:117] "RemoveContainer" containerID="67e80b6a35e0e515a36202a81c9fb53930fbd87c40253b4fc58b3895eac76fca" Sep 17 01:56:18.603470 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.603418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f8c450-731d-44de-a8a4-3b81a82e7ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.603549 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.603469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwtx\" (UniqueName: \"kubernetes.io/projected/77f8c450-731d-44de-a8a4-3b81a82e7ee6-kube-api-access-tpwtx\") pod \"payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.704375 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.704340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f8c450-731d-44de-a8a4-3b81a82e7ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.704503 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.704395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwtx\" (UniqueName: \"kubernetes.io/projected/77f8c450-731d-44de-a8a4-3b81a82e7ee6-kube-api-access-tpwtx\") pod \"payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.705001 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.704983 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f8c450-731d-44de-a8a4-3b81a82e7ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.712800 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.712776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwtx\" (UniqueName: \"kubernetes.io/projected/77f8c450-731d-44de-a8a4-3b81a82e7ee6-kube-api-access-tpwtx\") pod \"payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.734032 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.734011 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:56:18.795873 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.795850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:18.852550 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.852452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:18.907186 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907121 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-socket\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907186 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907163 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istiod-ca-cert\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907261 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-certs\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907289 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-envoy\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907320 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-token\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907350 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907345 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-podinfo\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-data\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907428 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-credential-socket\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907444 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:56:18.907507 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907478 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8srqw\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-kube-api-access-8srqw\") pod \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\" (UID: \"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4\") " Sep 17 01:56:18.907674 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907572 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:56:18.907674 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:18.907858 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907834 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:18.907858 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907860 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istiod-ca-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:18.908081 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.907873 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-workload-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:18.908081 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.908020 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:56:18.908201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.908098 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:56:18.909804 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.909784 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:56:18.909913 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.909894 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:18.910009 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.909991 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:56:18.910133 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:18.910120 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-kube-api-access-8srqw" (OuterVolumeSpecName: "kube-api-access-8srqw") pod "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" (UID: "8b17fa37-01a6-48ef-8ee7-1f3039e0bce4"). InnerVolumeSpecName "kube-api-access-8srqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:19.008568 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.008550 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-envoy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:19.008568 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.008569 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:19.008735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.008577 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-podinfo\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:19.008735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.008586 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-istio-data\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:19.008735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.008593 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-credential-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:19.008735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.008601 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srqw\" (UniqueName: \"kubernetes.io/projected/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4-kube-api-access-8srqw\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:19.065187 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.065167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p"] Sep 17 01:56:19.066709 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:19.066688 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f8c450_731d_44de_a8a4_3b81a82e7ee6.slice/crio-b7d33fa23f47870da1ec64bb51ab4e62d2dc4c0f03c514a1f90b18c12541ba7f WatchSource:0}: Error finding container b7d33fa23f47870da1ec64bb51ab4e62d2dc4c0f03c514a1f90b18c12541ba7f: Status 404 returned error can't find the container with id b7d33fa23f47870da1ec64bb51ab4e62d2dc4c0f03c514a1f90b18c12541ba7f Sep 17 01:56:19.530143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.529601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" event={"ID":"77f8c450-731d-44de-a8a4-3b81a82e7ee6","Type":"ContainerStarted","Data":"50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114"} Sep 17 01:56:19.530143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.529675 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:19.530143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.529699 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" event={"ID":"77f8c450-731d-44de-a8a4-3b81a82e7ee6","Type":"ContainerStarted","Data":"b7d33fa23f47870da1ec64bb51ab4e62d2dc4c0f03c514a1f90b18c12541ba7f"} Sep 17 01:56:19.531763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.531614 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" Sep 17 01:56:19.531763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.531630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6" event={"ID":"8b17fa37-01a6-48ef-8ee7-1f3039e0bce4","Type":"ContainerDied","Data":"540831c23386bf836592bfe96dd9b7ce22acc78633ab4e90b564194d00114dcd"} Sep 17 01:56:19.531763 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.531696 2582 scope.go:117] "RemoveContainer" containerID="55e000994b1844ac75a0ba47a5f9132418962573e786f73aa3fe596dc35f0611" Sep 17 01:56:19.547541 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.547489 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" podStartSLOduration=1.547470433 podStartE2EDuration="1.547470433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:19.544381554 +0000 UTC m=+1124.379433889" watchObservedRunningTime="2026-09-17 01:56:19.547470433 +0000 UTC m=+1124.382522766" Sep 17 01:56:19.561569 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.561543 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6"] Sep 17 01:56:19.567491 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.567470 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-266b61f5-openshift-default-5475564bdd-c4rc6"] Sep 17 01:56:19.589197 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.589171 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:19.589297 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.589219 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:19.601403 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.601387 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:19.621130 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.621099 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:19.678587 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:19.678560 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" path="/var/lib/kubelet/pods/8b17fa37-01a6-48ef-8ee7-1f3039e0bce4/volumes" Sep 17 01:56:20.455370 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:20.455334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:20.554789 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:20.554765 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 01:56:21.294584 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:21.294546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:25.987892 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:25.987857 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:56:30.538928 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:30.538897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:38.822554 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:38.822511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:40.444436 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:40.444391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:41.385147 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:41.385115 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:42.139441 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:42.139405 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:43.151201 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:43.151156 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:43.885930 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:43.885894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:44.609094 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.609053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:44.772314 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.772277 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw"] Sep 17 01:56:44.772651 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.772639 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" Sep 17 01:56:44.772695 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.772653 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" Sep 17 01:56:44.772734 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.772728 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" Sep 17 01:56:44.777169 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.777151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:44.779280 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.779256 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 01:56:44.779401 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.779382 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:56:44.783074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.783050 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw"] Sep 17 01:56:44.833409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.833366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a85309f-a7d1-48de-8994-26727ca060c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw\" (UID: \"7a85309f-a7d1-48de-8994-26727ca060c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:44.934227 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.934124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a85309f-a7d1-48de-8994-26727ca060c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw\" (UID: \"7a85309f-a7d1-48de-8994-26727ca060c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:44.934753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:44.934731 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a85309f-a7d1-48de-8994-26727ca060c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw\" (UID: \"7a85309f-a7d1-48de-8994-26727ca060c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:45.088770 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.088736 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:45.357886 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.357861 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw"] Sep 17 01:56:45.359914 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:45.359880 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a85309f_a7d1_48de_8994_26727ca060c1.slice/crio-fbf65d5963bb960629229d476cdefe78df90886b68598add1b32a55af1de77da WatchSource:0}: Error finding container fbf65d5963bb960629229d476cdefe78df90886b68598add1b32a55af1de77da: Status 404 returned error can't find the container with id fbf65d5963bb960629229d476cdefe78df90886b68598add1b32a55af1de77da Sep 17 01:56:45.379982 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.379959 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf"] Sep 17 01:56:45.380548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.380530 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" Sep 17 01:56:45.380598 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.380553 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" Sep 17 01:56:45.380677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.380666 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b17fa37-01a6-48ef-8ee7-1f3039e0bce4" containerName="istio-proxy" Sep 17 01:56:45.385176 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.385154 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.388589 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.388569 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-38758b\"/\"kube-root-ca.crt\"" Sep 17 01:56:45.388692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.388622 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-38758b\"/\"test-model-38758b-kserve-self-signed-certs\"" Sep 17 01:56:45.388692 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.388635 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-38758b\"/\"openshift-service-ca.crt\"" Sep 17 01:56:45.388788 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.388694 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-38758b\"/\"default-dockercfg-bpgh8\"" Sep 17 01:56:45.396065 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.396043 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf"] Sep 17 01:56:45.439319 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.439290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q56s\" (UniqueName: \"kubernetes.io/projected/aa27ee67-842a-4c03-94b2-3e16ed0dee03-kube-api-access-6q56s\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.439500 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.439431 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-home\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.439500 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.439465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tls-certs\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.439500 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.439495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-dshm\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.439646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.439528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tmp-dir\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.439646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.439592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-model-cache\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.456579 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.456548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:45.540985 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.540951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-home\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541126 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.540996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tls-certs\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541126 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-dshm\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541126 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tmp-dir\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541126 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-model-cache\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541363 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q56s\" (UniqueName: \"kubernetes.io/projected/aa27ee67-842a-4c03-94b2-3e16ed0dee03-kube-api-access-6q56s\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541417 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-home\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541758 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541683 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-model-cache\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.541758 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.541704 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tmp-dir\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.563972 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.563946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:45.613756 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.613698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-dshm\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.614179 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.614135 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tls-certs\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.614334 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.614311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q56s\" (UniqueName: \"kubernetes.io/projected/aa27ee67-842a-4c03-94b2-3e16ed0dee03-kube-api-access-6q56s\") pod \"test-model-38758b-kserve-c67bb48c5-2fgwf\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.664646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.664617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" event={"ID":"7a85309f-a7d1-48de-8994-26727ca060c1","Type":"ContainerStarted","Data":"4ce18cf6113861041e38132d12b2f3291220473062065099dc9fa293a2c0d4f4"} Sep 17 01:56:45.664781 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.664665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" event={"ID":"7a85309f-a7d1-48de-8994-26727ca060c1","Type":"ContainerStarted","Data":"fbf65d5963bb960629229d476cdefe78df90886b68598add1b32a55af1de77da"} Sep 17 01:56:45.681130 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.681074 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" podStartSLOduration=1.681055684 podStartE2EDuration="1.681055684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:45.678131452 +0000 UTC m=+1150.513183799" watchObservedRunningTime="2026-09-17 01:56:45.681055684 +0000 UTC m=+1150.516108020" Sep 17 01:56:45.707227 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.707181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:45.976925 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:45.976899 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf"] Sep 17 01:56:45.978951 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:45.978923 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa27ee67_842a_4c03_94b2_3e16ed0dee03.slice/crio-974929f13cff3f797ddfdd2ef28700719170bbe5cd738a5680701d468eceb47d WatchSource:0}: Error finding container 974929f13cff3f797ddfdd2ef28700719170bbe5cd738a5680701d468eceb47d: Status 404 returned error can't find the container with id 974929f13cff3f797ddfdd2ef28700719170bbe5cd738a5680701d468eceb47d Sep 17 01:56:46.181032 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.181001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:46.672022 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.671989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" event={"ID":"aa27ee67-842a-4c03-94b2-3e16ed0dee03","Type":"ContainerStarted","Data":"bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89"} Sep 17 01:56:46.672533 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.672028 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" event={"ID":"aa27ee67-842a-4c03-94b2-3e16ed0dee03","Type":"ContainerStarted","Data":"974929f13cff3f797ddfdd2ef28700719170bbe5cd738a5680701d468eceb47d"} Sep 17 01:56:46.673659 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.673623 2582 generic.go:356] "Generic (PLEG): container finished" podID="7a85309f-a7d1-48de-8994-26727ca060c1" containerID="4ce18cf6113861041e38132d12b2f3291220473062065099dc9fa293a2c0d4f4" exitCode=0 Sep 17 01:56:46.673783 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.673696 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" event={"ID":"7a85309f-a7d1-48de-8994-26727ca060c1","Type":"ContainerDied","Data":"4ce18cf6113861041e38132d12b2f3291220473062065099dc9fa293a2c0d4f4"} Sep 17 01:56:46.688477 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.688437 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" podStartSLOduration=1.688422343 podStartE2EDuration="1.688422343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:46.687068199 +0000 UTC m=+1151.522120542" watchObservedRunningTime="2026-09-17 01:56:46.688422343 +0000 UTC m=+1151.523474682" Sep 17 01:56:46.957521 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:46.957433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:47.733086 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:47.733001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:47.807433 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:47.807410 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:47.862299 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:47.862238 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a85309f-a7d1-48de-8994-26727ca060c1-maas-api-service-ca\") pod \"7a85309f-a7d1-48de-8994-26727ca060c1\" (UID: \"7a85309f-a7d1-48de-8994-26727ca060c1\") " Sep 17 01:56:47.862585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:47.862566 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a85309f-a7d1-48de-8994-26727ca060c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a85309f-a7d1-48de-8994-26727ca060c1" (UID: "7a85309f-a7d1-48de-8994-26727ca060c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:47.963704 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:47.963682 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a85309f-a7d1-48de-8994-26727ca060c1-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:48.539227 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:48.539181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:48.688153 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:48.688129 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" Sep 17 01:56:48.688153 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:48.688148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw" event={"ID":"7a85309f-a7d1-48de-8994-26727ca060c1","Type":"ContainerDied","Data":"fbf65d5963bb960629229d476cdefe78df90886b68598add1b32a55af1de77da"} Sep 17 01:56:48.688461 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:48.688185 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf65d5963bb960629229d476cdefe78df90886b68598add1b32a55af1de77da" Sep 17 01:56:49.347693 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:49.347663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:52.374719 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:52.374683 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:56:55.708177 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:55.708145 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:55.708575 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:55.708191 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:55.724286 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:55.724262 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:55.736315 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:55.736281 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 01:56:56.145180 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.145143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:56.179886 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.179855 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p"] Sep 17 01:56:56.180114 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.180084 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" podUID="77f8c450-731d-44de-a8a4-3b81a82e7ee6" containerName="payload-pre-processing" containerID="cri-o://50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114" gracePeriod=30 Sep 17 01:56:56.246584 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.246553 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:56.335648 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.335614 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw"] Sep 17 01:56:56.338273 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.338240 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:56.338426 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.338257 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-404126c3-5pmjw"] Sep 17 01:56:56.387547 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.387511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:56.424711 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.424693 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:56.498402 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.498372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:56.532146 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.532120 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f8c450-731d-44de-a8a4-3b81a82e7ee6-plugins-config-volume\") pod \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " Sep 17 01:56:56.532308 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.532280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpwtx\" (UniqueName: \"kubernetes.io/projected/77f8c450-731d-44de-a8a4-3b81a82e7ee6-kube-api-access-tpwtx\") pod \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\" (UID: \"77f8c450-731d-44de-a8a4-3b81a82e7ee6\") " Sep 17 01:56:56.532521 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.532500 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f8c450-731d-44de-a8a4-3b81a82e7ee6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77f8c450-731d-44de-a8a4-3b81a82e7ee6" (UID: "77f8c450-731d-44de-a8a4-3b81a82e7ee6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:56.534204 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.534184 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f8c450-731d-44de-a8a4-3b81a82e7ee6-kube-api-access-tpwtx" (OuterVolumeSpecName: "kube-api-access-tpwtx") pod "77f8c450-731d-44de-a8a4-3b81a82e7ee6" (UID: "77f8c450-731d-44de-a8a4-3b81a82e7ee6"). InnerVolumeSpecName "kube-api-access-tpwtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:56.632864 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.632836 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpwtx\" (UniqueName: \"kubernetes.io/projected/77f8c450-731d-44de-a8a4-3b81a82e7ee6-kube-api-access-tpwtx\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:56.632864 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.632860 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f8c450-731d-44de-a8a4-3b81a82e7ee6-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:56:56.713712 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.713684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:56.727291 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.727270 2582 generic.go:356] "Generic (PLEG): container finished" podID="77f8c450-731d-44de-a8a4-3b81a82e7ee6" containerID="50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114" exitCode=0 Sep 17 01:56:56.727409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.727326 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" Sep 17 01:56:56.727409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.727357 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" event={"ID":"77f8c450-731d-44de-a8a4-3b81a82e7ee6","Type":"ContainerDied","Data":"50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114"} Sep 17 01:56:56.727409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.727403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p" event={"ID":"77f8c450-731d-44de-a8a4-3b81a82e7ee6","Type":"ContainerDied","Data":"b7d33fa23f47870da1ec64bb51ab4e62d2dc4c0f03c514a1f90b18c12541ba7f"} Sep 17 01:56:56.727548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.727426 2582 scope.go:117] "RemoveContainer" containerID="50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114" Sep 17 01:56:56.736104 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.736089 2582 scope.go:117] "RemoveContainer" containerID="50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114" Sep 17 01:56:56.736396 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:56:56.736378 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114\": container with ID starting with 50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114 not found: ID does not exist" containerID="50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114" Sep 17 01:56:56.736453 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.736405 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114"} err="failed to get container status \"50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114\": rpc error: code = NotFound desc = could not find container \"50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114\": container with ID starting with 50bb1e29ea49fab6109358f3027350567c5fa350e2afc657594d4fff855a5114 not found: ID does not exist" Sep 17 01:56:56.748119 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.748100 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p"] Sep 17 01:56:56.751409 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.751387 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-404126c3-56944b8478-vxc2p"] Sep 17 01:56:56.999992 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:56.999912 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:57.306664 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.306588 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt"] Sep 17 01:56:57.307180 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.307165 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f8c450-731d-44de-a8a4-3b81a82e7ee6" containerName="payload-pre-processing" Sep 17 01:56:57.307240 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.307184 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f8c450-731d-44de-a8a4-3b81a82e7ee6" containerName="payload-pre-processing" Sep 17 01:56:57.307240 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.307199 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a85309f-a7d1-48de-8994-26727ca060c1" containerName="revoke-keys" Sep 17 01:56:57.307240 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.307222 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a85309f-a7d1-48de-8994-26727ca060c1" containerName="revoke-keys" Sep 17 01:56:57.307342 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.307304 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f8c450-731d-44de-a8a4-3b81a82e7ee6" containerName="payload-pre-processing" Sep 17 01:56:57.307342 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.307317 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a85309f-a7d1-48de-8994-26727ca060c1" containerName="revoke-keys" Sep 17 01:56:57.312018 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.311998 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.314448 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.314425 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3607043f-openshift-default-dockercfg-qb7n4\"" Sep 17 01:56:57.314594 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.314467 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 01:56:57.323494 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.323470 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt"] Sep 17 01:56:57.324844 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.324816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:57.418906 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.418881 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:57.438499 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-podinfo\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438593 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-socket\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438593 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-credential-socket\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-data\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57611b-094e-4387-a8d8-3ef7054a955e-istiod-ca-cert\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-certs\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438753 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-token\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438826 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x29n\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-kube-api-access-7x29n\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.438826 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.438787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-envoy\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539524 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-certs\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539524 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-token\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539690 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x29n\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-kube-api-access-7x29n\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539690 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-envoy\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539690 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-podinfo\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539690 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-socket\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-credential-socket\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-data\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.539865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57611b-094e-4387-a8d8-3ef7054a955e-istiod-ca-cert\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.540020 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.539915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-certs\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.540081 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.540071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-socket\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.540154 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.540134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-data\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.540228 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.540157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-credential-socket\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.540568 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.540551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57611b-094e-4387-a8d8-3ef7054a955e-istiod-ca-cert\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.541861 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.541843 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-envoy\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.542124 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.542106 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-podinfo\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.547388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.547366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-token\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.548120 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.548100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x29n\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-kube-api-access-7x29n\") pod \"e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.624646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.624587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:57.685196 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.685162 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f8c450-731d-44de-a8a4-3b81a82e7ee6" path="/var/lib/kubelet/pods/77f8c450-731d-44de-a8a4-3b81a82e7ee6/volumes" Sep 17 01:56:57.685699 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.685677 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a85309f-a7d1-48de-8994-26727ca060c1" path="/var/lib/kubelet/pods/7a85309f-a7d1-48de-8994-26727ca060c1/volumes" Sep 17 01:56:57.899876 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.899849 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt"] Sep 17 01:56:57.901870 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:56:57.901841 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b57611b_094e_4387_a8d8_3ef7054a955e.slice/crio-c776867bc863e769d85d8023e7b867c2b263ddfc81984bae4a81865821f3e123 WatchSource:0}: Error finding container c776867bc863e769d85d8023e7b867c2b263ddfc81984bae4a81865821f3e123: Status 404 returned error can't find the container with id c776867bc863e769d85d8023e7b867c2b263ddfc81984bae4a81865821f3e123 Sep 17 01:56:57.904248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.904185 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:56:57.904332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.904281 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:56:57.904332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:57.904317 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:56:58.082133 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:58.082081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:58.741506 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:58.741471 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" event={"ID":"1b57611b-094e-4387-a8d8-3ef7054a955e","Type":"ContainerStarted","Data":"5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656"} Sep 17 01:56:58.741680 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:58.741521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" event={"ID":"1b57611b-094e-4387-a8d8-3ef7054a955e","Type":"ContainerStarted","Data":"c776867bc863e769d85d8023e7b867c2b263ddfc81984bae4a81865821f3e123"} Sep 17 01:56:58.761858 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:58.761810 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podStartSLOduration=1.7617963749999999 podStartE2EDuration="1.761796375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:56:58.758972836 +0000 UTC m=+1163.594025169" watchObservedRunningTime="2026-09-17 01:56:58.761796375 +0000 UTC m=+1163.596848709" Sep 17 01:56:58.952185 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:58.952139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:56:59.624987 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:59.624949 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:59.630065 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:59.630043 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:59.746548 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:59.746526 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:59.747141 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:59.747124 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:56:59.808854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:56:59.808825 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:00.218180 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:00.218148 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:00.307110 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:00.307077 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:00.534140 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:00.534062 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:01.257876 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:01.257840 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:02.014930 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:02.014893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:02.709170 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:02.709138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:03.651108 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:03.651048 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:03.755790 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:03.755754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:03.816388 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:03.816281 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:04.200802 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:04.200702 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:04.296124 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:04.296086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:11.258891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.258856 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv"] Sep 17 01:57:11.262975 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.262955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.265403 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.265383 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"premium-w0-e564e1-kserve-self-signed-certs\"" Sep 17 01:57:11.268906 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.268886 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv"] Sep 17 01:57:11.282056 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.282027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:11.359638 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.359617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-model-cache\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.359751 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.359655 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1edbb2-d236-409a-82f9-6038b13b7970-tls-certs\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.359751 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.359687 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-dshm\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.359921 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.359757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-home\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.359921 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.359886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-tmp-dir\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.360008 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.359921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4vb\" (UniqueName: \"kubernetes.io/projected/ae1edbb2-d236-409a-82f9-6038b13b7970-kube-api-access-nt4vb\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.460731 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.460704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1edbb2-d236-409a-82f9-6038b13b7970-tls-certs\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.460854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.460748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-dshm\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.460854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.460780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-home\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.460854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.460799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-tmp-dir\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.460854 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.460814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4vb\" (UniqueName: \"kubernetes.io/projected/ae1edbb2-d236-409a-82f9-6038b13b7970-kube-api-access-nt4vb\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.461047 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.460997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-model-cache\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.461253 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.461232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-tmp-dir\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.461332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.461252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-home\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.461372 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.461355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-model-cache\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.463025 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.463004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-dshm\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.463124 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.463025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1edbb2-d236-409a-82f9-6038b13b7970-tls-certs\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.469251 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.469232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4vb\" (UniqueName: \"kubernetes.io/projected/ae1edbb2-d236-409a-82f9-6038b13b7970-kube-api-access-nt4vb\") pod \"premium-w0-e564e1-kserve-745c9764f4-fpvfv\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.573661 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.573587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:11.839810 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:11.839786 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv"] Sep 17 01:57:11.841821 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:57:11.841792 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1edbb2_d236_409a_82f9_6038b13b7970.slice/crio-e437d1d1a34d5be1a21c0a32f0829c5b8da28178dbdbb8300cb759b3c704222c WatchSource:0}: Error finding container e437d1d1a34d5be1a21c0a32f0829c5b8da28178dbdbb8300cb759b3c704222c: Status 404 returned error can't find the container with id e437d1d1a34d5be1a21c0a32f0829c5b8da28178dbdbb8300cb759b3c704222c Sep 17 01:57:12.810271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:12.810225 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" event={"ID":"ae1edbb2-d236-409a-82f9-6038b13b7970","Type":"ContainerStarted","Data":"d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0"} Sep 17 01:57:12.810271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:12.810279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" event={"ID":"ae1edbb2-d236-409a-82f9-6038b13b7970","Type":"ContainerStarted","Data":"e437d1d1a34d5be1a21c0a32f0829c5b8da28178dbdbb8300cb759b3c704222c"} Sep 17 01:57:12.833243 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:12.829046 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" podStartSLOduration=1.8290185509999999 podStartE2EDuration="1.829018551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:57:12.825865657 +0000 UTC m=+1177.660917989" watchObservedRunningTime="2026-09-17 01:57:12.829018551 +0000 UTC m=+1177.664070885" Sep 17 01:57:15.679091 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:15.679053 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:57:19.938499 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:19.938453 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:21.574501 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:21.574460 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:21.574930 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:21.574517 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:21.587379 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:21.587352 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:21.871032 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:21.870958 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 01:57:25.630889 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:25.630844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:26.339953 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.339910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:26.888568 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.888531 2582 generic.go:356] "Generic (PLEG): container finished" podID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerID="5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656" exitCode=0 Sep 17 01:57:26.888979 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.888601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" event={"ID":"1b57611b-094e-4387-a8d8-3ef7054a955e","Type":"ContainerDied","Data":"5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656"} Sep 17 01:57:26.889075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.889060 2582 scope.go:117] "RemoveContainer" containerID="5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656" Sep 17 01:57:26.890295 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.890264 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:57:26.890396 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.890341 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:57:26.890396 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:26.890371 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 01:57:27.036971 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:27.036934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:27.810542 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:27.810508 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:27.895658 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:27.895623 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" event={"ID":"1b57611b-094e-4387-a8d8-3ef7054a955e","Type":"ContainerStarted","Data":"2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf"} Sep 17 01:57:28.470705 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:28.470667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:28.625136 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:28.625094 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:57:28.626364 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:28.626328 2582 patch_prober.go:28] interesting pod/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 17 01:57:28.626520 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:28.626393 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 17 01:57:29.134443 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:29.134401 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:29.625792 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:29.625745 2582 patch_prober.go:28] interesting pod/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 17 01:57:29.626082 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:29.626055 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 17 01:57:29.747060 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:29.747026 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:57:30.625624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:30.625580 2582 patch_prober.go:28] interesting pod/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 17 01:57:30.626290 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:30.626251 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 17 01:57:31.120456 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.120417 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67"] Sep 17 01:57:31.126177 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.126152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:31.129238 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.129199 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 01:57:31.129697 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.129677 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:57:31.134733 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.134692 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67"] Sep 17 01:57:31.253536 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.253498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0300c317-daec-40d2-a083-d4ab87dae162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3607043f-z4f67\" (UID: \"0300c317-daec-40d2-a083-d4ab87dae162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:31.354048 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.354008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0300c317-daec-40d2-a083-d4ab87dae162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3607043f-z4f67\" (UID: \"0300c317-daec-40d2-a083-d4ab87dae162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:31.354731 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.354703 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0300c317-daec-40d2-a083-d4ab87dae162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3607043f-z4f67\" (UID: \"0300c317-daec-40d2-a083-d4ab87dae162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:31.445131 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.445040 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:31.632233 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.629502 2582 patch_prober.go:28] interesting pod/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 17 01:57:31.632233 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.629571 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 17 01:57:31.711813 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.711788 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67"] Sep 17 01:57:31.713320 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:57:31.713286 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0300c317_daec_40d2_a083_d4ab87dae162.slice/crio-58c6236e1835b312e3c5cbb4a16889826d9ba6ab295697fcbcf6078c3dd001d0 WatchSource:0}: Error finding container 58c6236e1835b312e3c5cbb4a16889826d9ba6ab295697fcbcf6078c3dd001d0: Status 404 returned error can't find the container with id 58c6236e1835b312e3c5cbb4a16889826d9ba6ab295697fcbcf6078c3dd001d0 Sep 17 01:57:31.917537 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.917458 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" event={"ID":"0300c317-daec-40d2-a083-d4ab87dae162","Type":"ContainerStarted","Data":"58c6236e1835b312e3c5cbb4a16889826d9ba6ab295697fcbcf6078c3dd001d0"} Sep 17 01:57:31.950043 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:31.950002 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:32.625366 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:32.625322 2582 patch_prober.go:28] interesting pod/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 17 01:57:32.625665 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:32.625617 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 17 01:57:32.643579 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:32.643545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:32.924512 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:32.924420 2582 generic.go:356] "Generic (PLEG): container finished" podID="0300c317-daec-40d2-a083-d4ab87dae162" containerID="a0aaf07243aa2db67692b47789eeefde3c09b9a1e4c8f127e9129ed99887dd50" exitCode=0 Sep 17 01:57:32.924672 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:32.924511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" event={"ID":"0300c317-daec-40d2-a083-d4ab87dae162","Type":"ContainerDied","Data":"a0aaf07243aa2db67692b47789eeefde3c09b9a1e4c8f127e9129ed99887dd50"} Sep 17 01:57:33.444610 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:33.444568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:33.628835 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:33.628807 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:57:33.629827 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:33.629805 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:57:34.055651 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.055630 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:34.126502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.126470 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:34.179419 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.179357 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0300c317-daec-40d2-a083-d4ab87dae162-maas-api-service-ca\") pod \"0300c317-daec-40d2-a083-d4ab87dae162\" (UID: \"0300c317-daec-40d2-a083-d4ab87dae162\") " Sep 17 01:57:34.179677 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.179656 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0300c317-daec-40d2-a083-d4ab87dae162-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0300c317-daec-40d2-a083-d4ab87dae162" (UID: "0300c317-daec-40d2-a083-d4ab87dae162"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:57:34.280531 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.280504 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0300c317-daec-40d2-a083-d4ab87dae162-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:34.776411 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.776381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:34.934645 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.934622 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" Sep 17 01:57:34.934810 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.934663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67" event={"ID":"0300c317-daec-40d2-a083-d4ab87dae162","Type":"ContainerDied","Data":"58c6236e1835b312e3c5cbb4a16889826d9ba6ab295697fcbcf6078c3dd001d0"} Sep 17 01:57:34.934810 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:34.934709 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c6236e1835b312e3c5cbb4a16889826d9ba6ab295697fcbcf6078c3dd001d0" Sep 17 01:57:35.550136 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:35.550098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:41.908318 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:41.908287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:41.980241 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:41.980189 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:42.042879 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:42.042842 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:42.110539 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:42.110502 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67"] Sep 17 01:57:42.112575 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:42.112546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:42.112798 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:42.112781 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3607043f-z4f67"] Sep 17 01:57:42.172946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:42.172858 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:42.548489 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:42.548453 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:43.258271 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.258235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:43.373799 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.373760 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt"] Sep 17 01:57:43.374113 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.374081 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" containerID="cri-o://2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf" gracePeriod=30 Sep 17 01:57:43.387592 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.387558 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:43.679472 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.679387 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0300c317-daec-40d2-a083-d4ab87dae162" path="/var/lib/kubelet/pods/0300c317-daec-40d2-a083-d4ab87dae162/volumes" Sep 17 01:57:43.850091 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.850057 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:43.950598 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:43.950510 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:44.181143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:44.181110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:44.356736 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:44.356703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:44.829563 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:44.829521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:45.528891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:45.528856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:46.308957 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:46.308924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:46.854641 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:46.854609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:46.945746 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:46.945712 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:47.307010 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:47.306980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:47.973130 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:47.973097 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:48.622835 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.622813 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:57:48.711186 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711164 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-credential-socket\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711316 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711195 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-podinfo\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711316 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711233 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-socket\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711316 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711252 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-token\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711439 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711392 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-envoy\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711509 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711486 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-certs\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711576 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711508 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:48.711576 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711526 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:48.711576 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711546 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x29n\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-kube-api-access-7x29n\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711730 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711623 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-data\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711730 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57611b-094e-4387-a8d8-3ef7054a955e-istiod-ca-cert\") pod \"1b57611b-094e-4387-a8d8-3ef7054a955e\" (UID: \"1b57611b-094e-4387-a8d8-3ef7054a955e\") " Sep 17 01:57:48.711841 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711733 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:48.711985 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.711904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:48.712099 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.712005 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-data\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.712178 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.712103 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-credential-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.712178 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.712119 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.712178 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.712133 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-workload-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.712776 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.712748 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b57611b-094e-4387-a8d8-3ef7054a955e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:57:48.713732 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.713699 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:48.714004 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.713982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:57:48.714081 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.713999 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-kube-api-access-7x29n" (OuterVolumeSpecName: "kube-api-access-7x29n") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "kube-api-access-7x29n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:48.714151 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.714122 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b57611b-094e-4387-a8d8-3ef7054a955e" (UID: "1b57611b-094e-4387-a8d8-3ef7054a955e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:48.743565 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.743541 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:48.812632 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.812611 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x29n\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-kube-api-access-7x29n\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.812632 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.812631 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b57611b-094e-4387-a8d8-3ef7054a955e-istiod-ca-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.812756 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.812642 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-podinfo\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.812756 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.812651 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:48.812756 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:48.812659 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b57611b-094e-4387-a8d8-3ef7054a955e-istio-envoy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:57:49.002332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.002244 2582 generic.go:356] "Generic (PLEG): container finished" podID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerID="2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf" exitCode=0 Sep 17 01:57:49.002332 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.002312 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" Sep 17 01:57:49.002759 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.002342 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" event={"ID":"1b57611b-094e-4387-a8d8-3ef7054a955e","Type":"ContainerDied","Data":"2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf"} Sep 17 01:57:49.002759 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.002400 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt" event={"ID":"1b57611b-094e-4387-a8d8-3ef7054a955e","Type":"ContainerDied","Data":"c776867bc863e769d85d8023e7b867c2b263ddfc81984bae4a81865821f3e123"} Sep 17 01:57:49.002759 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.002432 2582 scope.go:117] "RemoveContainer" containerID="2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf" Sep 17 01:57:49.011567 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.011550 2582 scope.go:117] "RemoveContainer" containerID="5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656" Sep 17 01:57:49.022075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.022052 2582 scope.go:117] "RemoveContainer" containerID="2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf" Sep 17 01:57:49.022415 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:57:49.022393 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf\": container with ID starting with 2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf not found: ID does not exist" containerID="2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf" Sep 17 01:57:49.022511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.022423 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf"} err="failed to get container status \"2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf\": rpc error: code = NotFound desc = could not find container \"2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf\": container with ID starting with 2ab88b00f85f00a002b1aaf7cccae7399f6dd9d10a0824923a61a87b2db257bf not found: ID does not exist" Sep 17 01:57:49.022511 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.022440 2582 scope.go:117] "RemoveContainer" containerID="5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656" Sep 17 01:57:49.022750 ip-10-0-142-18 kubenswrapper[2582]: E0917 01:57:49.022730 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656\": container with ID starting with 5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656 not found: ID does not exist" containerID="5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656" Sep 17 01:57:49.022812 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.022756 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656"} err="failed to get container status \"5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656\": rpc error: code = NotFound desc = could not find container \"5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656\": container with ID starting with 5cceeffd6cfcab24260a97c68809cb8bccde85ffbfdac78b178df664218d1656 not found: ID does not exist" Sep 17 01:57:49.024746 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.024722 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt"] Sep 17 01:57:49.028599 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.028565 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3607043f-openshift-default-7454c4b674-bxrgt"] Sep 17 01:57:49.158502 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158465 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp"] Sep 17 01:57:49.158904 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158890 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" Sep 17 01:57:49.158949 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158906 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" Sep 17 01:57:49.158949 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158917 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0300c317-daec-40d2-a083-d4ab87dae162" containerName="revoke-keys" Sep 17 01:57:49.158949 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158923 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0300c317-daec-40d2-a083-d4ab87dae162" containerName="revoke-keys" Sep 17 01:57:49.158949 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158940 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" Sep 17 01:57:49.158949 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.158946 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" Sep 17 01:57:49.159117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.159008 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0300c317-daec-40d2-a083-d4ab87dae162" containerName="revoke-keys" Sep 17 01:57:49.159117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.159019 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" Sep 17 01:57:49.159117 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.159028 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" containerName="istio-proxy" Sep 17 01:57:49.162476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.162460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.164779 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.164760 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3f027f\"/\"premium-w3-3f027f-kserve-self-signed-certs\"" Sep 17 01:57:49.171254 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.171230 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp"] Sep 17 01:57:49.318155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.318043 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj9l5\" (UniqueName: \"kubernetes.io/projected/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-kube-api-access-mj9l5\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.318155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.318099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tmp-dir\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.318155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.318122 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-home\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.318155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.318141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-model-cache\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.318155 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.318160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tls-certs\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.318605 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.318255 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-dshm\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419038 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.418984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tmp-dir\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419038 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-home\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419373 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-model-cache\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419373 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tls-certs\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419373 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-dshm\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419373 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj9l5\" (UniqueName: \"kubernetes.io/projected/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-kube-api-access-mj9l5\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tmp-dir\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419509 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-home\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.419585 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.419568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-model-cache\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.421679 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.421653 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-dshm\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.422158 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.422023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tls-certs\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.428509 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.428409 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj9l5\" (UniqueName: \"kubernetes.io/projected/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-kube-api-access-mj9l5\") pod \"premium-w3-3f027f-kserve-7c765c7955-tkjhp\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.457112 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.457072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:49.472116 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.472078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:49.559866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.559825 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:49.578294 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.578187 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:57:49.679868 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.679830 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b57611b-094e-4387-a8d8-3ef7054a955e" path="/var/lib/kubelet/pods/1b57611b-094e-4387-a8d8-3ef7054a955e/volumes" Sep 17 01:57:49.746363 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:49.746338 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp"] Sep 17 01:57:49.748374 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:57:49.748349 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4cdd27c_0bef_4906_9e95_22a98d3bfa00.slice/crio-aa732318b407efe226ee60eca36d9d6e72b1a1057e2be2f925687ec25370baa1 WatchSource:0}: Error finding container aa732318b407efe226ee60eca36d9d6e72b1a1057e2be2f925687ec25370baa1: Status 404 returned error can't find the container with id aa732318b407efe226ee60eca36d9d6e72b1a1057e2be2f925687ec25370baa1 Sep 17 01:57:50.010971 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.010935 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" event={"ID":"f4cdd27c-0bef-4906-9e95-22a98d3bfa00","Type":"ContainerStarted","Data":"061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7"} Sep 17 01:57:50.011435 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.010979 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" event={"ID":"f4cdd27c-0bef-4906-9e95-22a98d3bfa00","Type":"ContainerStarted","Data":"aa732318b407efe226ee60eca36d9d6e72b1a1057e2be2f925687ec25370baa1"} Sep 17 01:57:50.030261 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.030182 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" podStartSLOduration=1.030159651 podStartE2EDuration="1.030159651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:57:50.028424659 +0000 UTC m=+1214.863477019" watchObservedRunningTime="2026-09-17 01:57:50.030159651 +0000 UTC m=+1214.865211985" Sep 17 01:57:50.260270 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.260225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:50.323830 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.323738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:50.423636 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.423591 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:50.818531 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.818487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:50.921873 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:50.921825 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:57:59.472859 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:59.472824 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:59.473423 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:59.472880 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:57:59.485194 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:57:59.485167 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:58:00.069321 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:00.069291 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 01:58:01.229821 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:01.229791 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:09.265715 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:09.265682 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:58:12.536387 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:12.536346 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:13.490673 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:13.490629 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:14.238846 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.238808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:14.270826 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.270792 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h"] Sep 17 01:58:14.273668 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.273638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.276946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.276922 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-55c648\"/\"default-dockercfg-zq7q7\"" Sep 17 01:58:14.276946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.276938 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-55c648\"/\"openshift-service-ca.crt\"" Sep 17 01:58:14.277161 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.276978 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-55c648\"/\"kube-root-ca.crt\"" Sep 17 01:58:14.277284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.277229 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-55c648\"/\"test-model-55c648-kserve-self-signed-certs\"" Sep 17 01:58:14.283354 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.283247 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h"] Sep 17 01:58:14.336650 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.336626 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lpv\" (UniqueName: \"kubernetes.io/projected/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-kube-api-access-64lpv\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.336755 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.336668 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-home\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.336755 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.336686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tls-certs\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.336852 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.336792 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-dshm\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.336852 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.336841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-model-cache\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.336937 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.336884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tmp-dir\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.338703 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.338671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:14.437663 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.437638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tmp-dir\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.437755 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.437680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lpv\" (UniqueName: \"kubernetes.io/projected/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-kube-api-access-64lpv\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.437755 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.437713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-home\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.437755 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.437728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tls-certs\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.437890 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.437758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-dshm\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.437890 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.437849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-model-cache\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.438067 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.438045 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tmp-dir\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.438116 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.438097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-home\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.438186 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.438168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-model-cache\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.440025 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.440002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-dshm\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.440231 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.440200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tls-certs\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.444866 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.444847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lpv\" (UniqueName: \"kubernetes.io/projected/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-kube-api-access-64lpv\") pod \"test-model-55c648-kserve-9d8869f67-5tt7h\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.585376 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.585326 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:14.854275 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:14.854252 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h"] Sep 17 01:58:14.856426 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:58:14.856402 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb53e2b5f_16cb_494b_af75_d6d0e1b9b18a.slice/crio-7ac26999bd6e37988a53ec19666c55d2da6acdc57b24bf44932cc2ca90d19a38 WatchSource:0}: Error finding container 7ac26999bd6e37988a53ec19666c55d2da6acdc57b24bf44932cc2ca90d19a38: Status 404 returned error can't find the container with id 7ac26999bd6e37988a53ec19666c55d2da6acdc57b24bf44932cc2ca90d19a38 Sep 17 01:58:15.134259 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:15.134162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" event={"ID":"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a","Type":"ContainerStarted","Data":"dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807"} Sep 17 01:58:15.134259 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:15.134200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" event={"ID":"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a","Type":"ContainerStarted","Data":"7ac26999bd6e37988a53ec19666c55d2da6acdc57b24bf44932cc2ca90d19a38"} Sep 17 01:58:15.150803 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:15.150759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" podStartSLOduration=1.150745511 podStartE2EDuration="1.150745511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:58:15.148857642 +0000 UTC m=+1239.983909976" watchObservedRunningTime="2026-09-17 01:58:15.150745511 +0000 UTC m=+1239.985797896" Sep 17 01:58:15.251284 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:15.251246 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:16.051133 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.051096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:16.902248 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.902200 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n"] Sep 17 01:58:16.905476 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.905459 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:16.907735 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.907715 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:58:16.907806 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.907737 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 01:58:16.913420 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.913401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n"] Sep 17 01:58:16.925262 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.925238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:16.962364 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:16.962339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f6ff2a-e242-4ead-867c-c3016b85cc57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-56574e4f-9h29n\" (UID: \"27f6ff2a-e242-4ead-867c-c3016b85cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:17.063801 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:17.063778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f6ff2a-e242-4ead-867c-c3016b85cc57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-56574e4f-9h29n\" (UID: \"27f6ff2a-e242-4ead-867c-c3016b85cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:17.064396 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:17.064376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f6ff2a-e242-4ead-867c-c3016b85cc57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-56574e4f-9h29n\" (UID: \"27f6ff2a-e242-4ead-867c-c3016b85cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:17.218301 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:17.218280 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:17.511254 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:17.511227 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n"] Sep 17 01:58:17.512768 ip-10-0-142-18 kubenswrapper[2582]: W0917 01:58:17.512734 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f6ff2a_e242_4ead_867c_c3016b85cc57.slice/crio-6f1e8c883d3c0687c5bad815ac3e98412a2bff7ae7df44aca1ff03d351d2281e WatchSource:0}: Error finding container 6f1e8c883d3c0687c5bad815ac3e98412a2bff7ae7df44aca1ff03d351d2281e: Status 404 returned error can't find the container with id 6f1e8c883d3c0687c5bad815ac3e98412a2bff7ae7df44aca1ff03d351d2281e Sep 17 01:58:17.835937 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:17.835912 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:18.151865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:18.151793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" event={"ID":"27f6ff2a-e242-4ead-867c-c3016b85cc57","Type":"ContainerStarted","Data":"ffa445f13f379dcf34ea31a6d51d35bcd42d0ffda1f74eef881da3bc0d77ecc8"} Sep 17 01:58:18.151865 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:18.151844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" event={"ID":"27f6ff2a-e242-4ead-867c-c3016b85cc57","Type":"ContainerStarted","Data":"6f1e8c883d3c0687c5bad815ac3e98412a2bff7ae7df44aca1ff03d351d2281e"} Sep 17 01:58:18.166340 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:18.166300 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" podStartSLOduration=2.166286342 podStartE2EDuration="2.166286342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:58:18.164616984 +0000 UTC m=+1242.999669318" watchObservedRunningTime="2026-09-17 01:58:18.166286342 +0000 UTC m=+1243.001338702" Sep 17 01:58:18.376948 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:18.376912 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:19.064223 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:19.064177 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:19.158893 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:19.158862 2582 generic.go:356] "Generic (PLEG): container finished" podID="27f6ff2a-e242-4ead-867c-c3016b85cc57" containerID="ffa445f13f379dcf34ea31a6d51d35bcd42d0ffda1f74eef881da3bc0d77ecc8" exitCode=0 Sep 17 01:58:19.159266 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:19.158951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" event={"ID":"27f6ff2a-e242-4ead-867c-c3016b85cc57","Type":"ContainerDied","Data":"ffa445f13f379dcf34ea31a6d51d35bcd42d0ffda1f74eef881da3bc0d77ecc8"} Sep 17 01:58:19.766918 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:19.766886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:20.297546 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:20.297519 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:20.393745 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:20.393716 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f6ff2a-e242-4ead-867c-c3016b85cc57-maas-api-service-ca\") pod \"27f6ff2a-e242-4ead-867c-c3016b85cc57\" (UID: \"27f6ff2a-e242-4ead-867c-c3016b85cc57\") " Sep 17 01:58:20.394044 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:20.394022 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f6ff2a-e242-4ead-867c-c3016b85cc57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27f6ff2a-e242-4ead-867c-c3016b85cc57" (UID: "27f6ff2a-e242-4ead-867c-c3016b85cc57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:58:20.494275 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:20.494253 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f6ff2a-e242-4ead-867c-c3016b85cc57-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 01:58:20.644050 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:20.644019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:21.170603 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:21.170575 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" Sep 17 01:58:21.170791 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:21.170614 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n" event={"ID":"27f6ff2a-e242-4ead-867c-c3016b85cc57","Type":"ContainerDied","Data":"6f1e8c883d3c0687c5bad815ac3e98412a2bff7ae7df44aca1ff03d351d2281e"} Sep 17 01:58:21.170791 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:21.170661 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f1e8c883d3c0687c5bad815ac3e98412a2bff7ae7df44aca1ff03d351d2281e" Sep 17 01:58:21.307051 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:21.307015 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:24.586106 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:24.586070 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:24.586106 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:24.586115 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:24.598023 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:24.597998 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:25.211884 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:25.211858 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 01:58:27.184257 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:27.184223 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:58:28.713334 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.713251 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:28.784708 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.784671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:28.841310 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.841279 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:28.916659 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.916632 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n"] Sep 17 01:58:28.918075 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.918033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:28.918768 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.918749 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56574e4f-9h29n"] Sep 17 01:58:28.966969 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:28.966890 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:29.083891 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:29.083851 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:29.413384 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:29.413301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:29.679481 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:29.679408 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f6ff2a-e242-4ead-867c-c3016b85cc57" path="/var/lib/kubelet/pods/27f6ff2a-e242-4ead-867c-c3016b85cc57/volumes" Sep 17 01:58:29.816494 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:29.816460 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:29.917143 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:29.917107 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:30.017467 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:30.017435 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:30.117093 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:30.117066 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:30.776833 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:30.776803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:31.471090 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:31.471059 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:31.571888 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:31.571860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:32.168803 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:32.168771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:35.867697 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:35.867661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:35.969515 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:35.969488 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:37.319752 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:37.319717 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:37.379565 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:37.379532 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:37.443347 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:37.443322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:37.544244 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:37.544196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:37.646371 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:37.646310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:38.045941 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:38.045908 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:58:55.868459 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:55.868424 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:58:57.218266 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:58:57.218228 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:03.091949 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:03.091917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:04.008751 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:04.008719 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:09.643459 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:09.643418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:09.708126 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:09.708098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:09.790266 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:09.790190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:09.892646 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:09.892607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:23.469751 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:23.469710 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:59:25.786084 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:25.786048 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:26.052074 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:26.051994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:30.604323 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:30.604289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:31.418499 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:31.418464 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:31.481603 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:31.481572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:31.561555 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:31.561526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:31.654376 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:31.654341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:38.772626 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:38.772590 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:59:42.175924 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:42.175892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:43.078374 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:43.078337 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:46.998572 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:46.998539 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf"] Sep 17 01:59:47.016395 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:47.016363 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:48.643471 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:48.643439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:48.748144 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:48.748109 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:48.845133 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:48.845103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:48.969946 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:48.969921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:49.629323 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:49.629290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:49.735265 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:49.735238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:49.827360 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:49.827331 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:51.369115 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:51.369079 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:57.087007 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:57.086974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:57.186139 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:57.186102 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:57.577624 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:57.577594 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 01:59:57.806203 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:57.806166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 01:59:59.363990 ip-10-0-142-18 kubenswrapper[2582]: I0917 01:59:59.363942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:00.139661 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.139621 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs"] Sep 17 02:00:00.140016 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.139996 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f6ff2a-e242-4ead-867c-c3016b85cc57" containerName="revoke-keys" Sep 17 02:00:00.140016 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.140011 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f6ff2a-e242-4ead-867c-c3016b85cc57" containerName="revoke-keys" Sep 17 02:00:00.140171 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.140071 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f6ff2a-e242-4ead-867c-c3016b85cc57" containerName="revoke-keys" Sep 17 02:00:00.143304 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.143286 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g"] Sep 17 02:00:00.143414 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.143395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" Sep 17 02:00:00.145723 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.145705 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:00:00.146358 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.146342 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn"] Sep 17 02:00:00.146484 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.146463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" Sep 17 02:00:00.149345 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.149328 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" Sep 17 02:00:00.152467 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.152446 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs"] Sep 17 02:00:00.155376 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.155356 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" Sep 17 02:00:00.156767 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.156747 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn"] Sep 17 02:00:00.159037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.158861 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g"] Sep 17 02:00:00.166363 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.166332 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" Sep 17 02:00:00.201868 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.201842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" Sep 17 02:00:00.434377 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.434354 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs"] Sep 17 02:00:00.436125 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:00.436098 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a21c85_8f4b_42d5_910f_2c802562962a.slice/crio-9d66c83f92051455376e24ea7c02010159cf59a33c9808024a78f90a921f6a5b WatchSource:0}: Error finding container 9d66c83f92051455376e24ea7c02010159cf59a33c9808024a78f90a921f6a5b: Status 404 returned error can't find the container with id 9d66c83f92051455376e24ea7c02010159cf59a33c9808024a78f90a921f6a5b Sep 17 02:00:00.456374 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.456352 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g"] Sep 17 02:00:00.457288 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:00.457265 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7776039_a619_4752_89fe_78ceef3eff11.slice/crio-78c562f867b74afa670f1a3e668a6b5cb9c9d23ca16c669fcffbeef8f4d2824c WatchSource:0}: Error finding container 78c562f867b74afa670f1a3e668a6b5cb9c9d23ca16c669fcffbeef8f4d2824c: Status 404 returned error can't find the container with id 78c562f867b74afa670f1a3e668a6b5cb9c9d23ca16c669fcffbeef8f4d2824c Sep 17 02:00:00.502509 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.502485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn"] Sep 17 02:00:00.503783 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:00.503760 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93c57a7_eee5_4462_ac76_2fecb23ec1e1.slice/crio-d72576dbe73a04be34fa207f04ab8a06de3fb9711e9e107944fb16c503b71d77 WatchSource:0}: Error finding container d72576dbe73a04be34fa207f04ab8a06de3fb9711e9e107944fb16c503b71d77: Status 404 returned error can't find the container with id d72576dbe73a04be34fa207f04ab8a06de3fb9711e9e107944fb16c503b71d77 Sep 17 02:00:00.701022 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.700969 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" event={"ID":"a93c57a7-eee5-4462-ac76-2fecb23ec1e1","Type":"ContainerStarted","Data":"d72576dbe73a04be34fa207f04ab8a06de3fb9711e9e107944fb16c503b71d77"} Sep 17 02:00:00.702528 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.702506 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" event={"ID":"a7776039-a619-4752-89fe-78ceef3eff11","Type":"ContainerStarted","Data":"7ecb00685c4e1ca4c751e7ceb61ea0ea070124ea4f1c5020233f87208d2937ee"} Sep 17 02:00:00.702634 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.702549 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" event={"ID":"a7776039-a619-4752-89fe-78ceef3eff11","Type":"ContainerStarted","Data":"78c562f867b74afa670f1a3e668a6b5cb9c9d23ca16c669fcffbeef8f4d2824c"} Sep 17 02:00:00.703974 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.703949 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" event={"ID":"97a21c85-8f4b-42d5-910f-2c802562962a","Type":"ContainerStarted","Data":"7fcc71f83af3bb55bbeb767b6f5865d3fd6613a2e7e0018abfd58f59e76f75fe"} Sep 17 02:00:00.704057 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.703989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" event={"ID":"97a21c85-8f4b-42d5-910f-2c802562962a","Type":"ContainerStarted","Data":"9d66c83f92051455376e24ea7c02010159cf59a33c9808024a78f90a921f6a5b"} Sep 17 02:00:00.719369 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.719330 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" podStartSLOduration=0.719315866 podStartE2EDuration="719.315866ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:00.7168 +0000 UTC m=+1345.551852334" watchObservedRunningTime="2026-09-17 02:00:00.719315866 +0000 UTC m=+1345.554368200" Sep 17 02:00:00.731639 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:00.731597 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" podStartSLOduration=0.731580078 podStartE2EDuration="731.580078ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:00.730839944 +0000 UTC m=+1345.565892276" watchObservedRunningTime="2026-09-17 02:00:00.731580078 +0000 UTC m=+1345.566632429" Sep 17 02:00:01.710749 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:01.710678 2582 generic.go:356] "Generic (PLEG): container finished" podID="a93c57a7-eee5-4462-ac76-2fecb23ec1e1" containerID="8b7ba9671bbf19eece28f2c8100961955c23b72df0916af777f974b20245bd4b" exitCode=0 Sep 17 02:00:01.711104 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:01.710745 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" event={"ID":"a93c57a7-eee5-4462-ac76-2fecb23ec1e1","Type":"ContainerDied","Data":"8b7ba9671bbf19eece28f2c8100961955c23b72df0916af777f974b20245bd4b"} Sep 17 02:00:01.712149 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:01.712128 2582 generic.go:356] "Generic (PLEG): container finished" podID="a7776039-a619-4752-89fe-78ceef3eff11" containerID="7ecb00685c4e1ca4c751e7ceb61ea0ea070124ea4f1c5020233f87208d2937ee" exitCode=0 Sep 17 02:00:01.712265 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:01.712194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" event={"ID":"a7776039-a619-4752-89fe-78ceef3eff11","Type":"ContainerDied","Data":"7ecb00685c4e1ca4c751e7ceb61ea0ea070124ea4f1c5020233f87208d2937ee"} Sep 17 02:00:01.713801 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:01.713763 2582 generic.go:356] "Generic (PLEG): container finished" podID="97a21c85-8f4b-42d5-910f-2c802562962a" containerID="7fcc71f83af3bb55bbeb767b6f5865d3fd6613a2e7e0018abfd58f59e76f75fe" exitCode=0 Sep 17 02:00:01.713801 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:01.713787 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" event={"ID":"97a21c85-8f4b-42d5-910f-2c802562962a","Type":"ContainerDied","Data":"7fcc71f83af3bb55bbeb767b6f5865d3fd6613a2e7e0018abfd58f59e76f75fe"} Sep 17 02:00:02.019280 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.019237 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" podUID="aa27ee67-842a-4c03-94b2-3e16ed0dee03" containerName="main" containerID="cri-o://bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89" gracePeriod=45 Sep 17 02:00:02.273310 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.273242 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 02:00:02.341085 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341061 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q56s\" (UniqueName: \"kubernetes.io/projected/aa27ee67-842a-4c03-94b2-3e16ed0dee03-kube-api-access-6q56s\") pod \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " Sep 17 02:00:02.341194 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341091 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-dshm\") pod \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " Sep 17 02:00:02.341194 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341126 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-home\") pod \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " Sep 17 02:00:02.341194 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341162 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tls-certs\") pod \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " Sep 17 02:00:02.341356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341241 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tmp-dir\") pod \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " Sep 17 02:00:02.341356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-model-cache\") pod \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\" (UID: \"aa27ee67-842a-4c03-94b2-3e16ed0dee03\") " Sep 17 02:00:02.341475 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341453 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-home" (OuterVolumeSpecName: "home") pod "aa27ee67-842a-4c03-94b2-3e16ed0dee03" (UID: "aa27ee67-842a-4c03-94b2-3e16ed0dee03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:02.341562 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341537 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa27ee67-842a-4c03-94b2-3e16ed0dee03" (UID: "aa27ee67-842a-4c03-94b2-3e16ed0dee03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:02.341623 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341593 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:02.341669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.341628 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa27ee67-842a-4c03-94b2-3e16ed0dee03" (UID: "aa27ee67-842a-4c03-94b2-3e16ed0dee03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:02.343262 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.343242 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-dshm" (OuterVolumeSpecName: "dshm") pod "aa27ee67-842a-4c03-94b2-3e16ed0dee03" (UID: "aa27ee67-842a-4c03-94b2-3e16ed0dee03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:02.343636 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.343616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa27ee67-842a-4c03-94b2-3e16ed0dee03-kube-api-access-6q56s" (OuterVolumeSpecName: "kube-api-access-6q56s") pod "aa27ee67-842a-4c03-94b2-3e16ed0dee03" (UID: "aa27ee67-842a-4c03-94b2-3e16ed0dee03"). InnerVolumeSpecName "kube-api-access-6q56s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:02.343636 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.343626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa27ee67-842a-4c03-94b2-3e16ed0dee03" (UID: "aa27ee67-842a-4c03-94b2-3e16ed0dee03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:00:02.442046 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.442025 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q56s\" (UniqueName: \"kubernetes.io/projected/aa27ee67-842a-4c03-94b2-3e16ed0dee03-kube-api-access-6q56s\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:02.442046 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.442045 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:02.442172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.442055 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:02.442172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.442065 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:02.442172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.442074 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa27ee67-842a-4c03-94b2-3e16ed0dee03-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:02.720516 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.720490 2582 generic.go:356] "Generic (PLEG): container finished" podID="aa27ee67-842a-4c03-94b2-3e16ed0dee03" containerID="bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89" exitCode=0 Sep 17 02:00:02.720920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.720526 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" Sep 17 02:00:02.720920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.720579 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" event={"ID":"aa27ee67-842a-4c03-94b2-3e16ed0dee03","Type":"ContainerDied","Data":"bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89"} Sep 17 02:00:02.720920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.720621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf" event={"ID":"aa27ee67-842a-4c03-94b2-3e16ed0dee03","Type":"ContainerDied","Data":"974929f13cff3f797ddfdd2ef28700719170bbe5cd738a5680701d468eceb47d"} Sep 17 02:00:02.720920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.720648 2582 scope.go:117] "RemoveContainer" containerID="bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89" Sep 17 02:00:02.734386 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.734370 2582 scope.go:117] "RemoveContainer" containerID="bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89" Sep 17 02:00:02.734712 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:00:02.734684 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89\": container with ID starting with bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89 not found: ID does not exist" containerID="bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89" Sep 17 02:00:02.734783 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.734723 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89"} err="failed to get container status \"bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89\": rpc error: code = NotFound desc = could not find container \"bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89\": container with ID starting with bfd8675d67f941484b90577a3dc68713ae8633834747280411ea8ffe98074b89 not found: ID does not exist" Sep 17 02:00:02.747511 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.747459 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf"] Sep 17 02:00:02.749330 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.749305 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-38758b/test-model-38758b-kserve-c67bb48c5-2fgwf"] Sep 17 02:00:02.880761 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.880729 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" Sep 17 02:00:02.888867 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.888829 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" Sep 17 02:00:02.908153 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:02.908128 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" Sep 17 02:00:03.058592 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.058514 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h"] Sep 17 02:00:03.194530 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194504 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw"] Sep 17 02:00:03.194946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194928 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c57a7-eee5-4462-ac76-2fecb23ec1e1" containerName="cleanup" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194949 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c57a7-eee5-4462-ac76-2fecb23ec1e1" containerName="cleanup" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194967 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7776039-a619-4752-89fe-78ceef3eff11" containerName="cleanup" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194975 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7776039-a619-4752-89fe-78ceef3eff11" containerName="cleanup" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194985 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a21c85-8f4b-42d5-910f-2c802562962a" containerName="cleanup" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.194994 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a21c85-8f4b-42d5-910f-2c802562962a" containerName="cleanup" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.195009 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa27ee67-842a-4c03-94b2-3e16ed0dee03" containerName="main" Sep 17 02:00:03.195037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.195017 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa27ee67-842a-4c03-94b2-3e16ed0dee03" containerName="main" Sep 17 02:00:03.195394 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.195119 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a21c85-8f4b-42d5-910f-2c802562962a" containerName="cleanup" Sep 17 02:00:03.195394 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.195132 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93c57a7-eee5-4462-ac76-2fecb23ec1e1" containerName="cleanup" Sep 17 02:00:03.195394 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.195144 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7776039-a619-4752-89fe-78ceef3eff11" containerName="cleanup" Sep 17 02:00:03.195394 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.195155 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa27ee67-842a-4c03-94b2-3e16ed0dee03" containerName="main" Sep 17 02:00:03.198382 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.198357 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:03.200781 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.200762 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:00:03.204317 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.204299 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw"] Sep 17 02:00:03.250829 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.250796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54b585e-343c-4e52-9c85-8c15141d7760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw\" (UID: \"c54b585e-343c-4e52-9c85-8c15141d7760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:03.351996 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.351939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54b585e-343c-4e52-9c85-8c15141d7760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw\" (UID: \"c54b585e-343c-4e52-9c85-8c15141d7760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:03.352548 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.352529 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54b585e-343c-4e52-9c85-8c15141d7760-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw\" (UID: \"c54b585e-343c-4e52-9c85-8c15141d7760\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:03.396921 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.396892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:03.512283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.512260 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:03.539910 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.539866 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:03.680542 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.680483 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa27ee67-842a-4c03-94b2-3e16ed0dee03" path="/var/lib/kubelet/pods/aa27ee67-842a-4c03-94b2-3e16ed0dee03/volumes" Sep 17 02:00:03.727395 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.727373 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" Sep 17 02:00:03.727751 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.727372 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-slwpn" event={"ID":"a93c57a7-eee5-4462-ac76-2fecb23ec1e1","Type":"ContainerDied","Data":"d72576dbe73a04be34fa207f04ab8a06de3fb9711e9e107944fb16c503b71d77"} Sep 17 02:00:03.727751 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.727486 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d72576dbe73a04be34fa207f04ab8a06de3fb9711e9e107944fb16c503b71d77" Sep 17 02:00:03.728824 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.728807 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" Sep 17 02:00:03.728932 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.728845 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g" event={"ID":"a7776039-a619-4752-89fe-78ceef3eff11","Type":"ContainerDied","Data":"78c562f867b74afa670f1a3e668a6b5cb9c9d23ca16c669fcffbeef8f4d2824c"} Sep 17 02:00:03.728932 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.728885 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78c562f867b74afa670f1a3e668a6b5cb9c9d23ca16c669fcffbeef8f4d2824c" Sep 17 02:00:03.730401 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.730384 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" Sep 17 02:00:03.730507 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.730402 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs" event={"ID":"97a21c85-8f4b-42d5-910f-2c802562962a","Type":"ContainerDied","Data":"9d66c83f92051455376e24ea7c02010159cf59a33c9808024a78f90a921f6a5b"} Sep 17 02:00:03.730507 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.730439 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d66c83f92051455376e24ea7c02010159cf59a33c9808024a78f90a921f6a5b" Sep 17 02:00:03.781339 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:03.781321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw"] Sep 17 02:00:03.782745 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:03.782725 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54b585e_343c_4e52_9c85_8c15141d7760.slice/crio-83854e037251c5ea70667ab2067f8a50aac727428f4bb3372c1ea1f03d07aa10 WatchSource:0}: Error finding container 83854e037251c5ea70667ab2067f8a50aac727428f4bb3372c1ea1f03d07aa10: Status 404 returned error can't find the container with id 83854e037251c5ea70667ab2067f8a50aac727428f4bb3372c1ea1f03d07aa10 Sep 17 02:00:04.132723 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.132696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:04.612300 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.610343 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:04.700645 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.700605 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:04.702056 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.702035 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2"] Sep 17 02:00:04.705584 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.705568 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.708013 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.707989 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 02:00:04.708109 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.707989 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a1f5690-openshift-default-dockercfg-q8qwq\"" Sep 17 02:00:04.714822 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.714803 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2"] Sep 17 02:00:04.738190 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.738163 2582 generic.go:356] "Generic (PLEG): container finished" podID="c54b585e-343c-4e52-9c85-8c15141d7760" containerID="c5ceff9dc48c0f9e5d70c1d5b953b374a42bb1c6ccf9a8d7e5ea26f1b683e1d9" exitCode=0 Sep 17 02:00:04.738576 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.738229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" event={"ID":"c54b585e-343c-4e52-9c85-8c15141d7760","Type":"ContainerDied","Data":"c5ceff9dc48c0f9e5d70c1d5b953b374a42bb1c6ccf9a8d7e5ea26f1b683e1d9"} Sep 17 02:00:04.738576 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.738269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" event={"ID":"c54b585e-343c-4e52-9c85-8c15141d7760","Type":"ContainerStarted","Data":"83854e037251c5ea70667ab2067f8a50aac727428f4bb3372c1ea1f03d07aa10"} Sep 17 02:00:04.763220 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763193 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-credential-socket\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763322 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-envoy\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763322 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-podinfo\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763441 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istiod-ca-cert\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763441 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-socket\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763555 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-certs\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763555 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctrb\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-kube-api-access-wctrb\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763668 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-token\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.763668 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.763631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-data\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctrb\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-kube-api-access-wctrb\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-token\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-data\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864334 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-credential-socket\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-envoy\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-podinfo\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864448 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istiod-ca-cert\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-socket\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-certs\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864836 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-data\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.864836 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.864732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-credential-socket\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.865039 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.865011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-certs\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.865205 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.865187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-socket\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.865278 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.865265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istiod-ca-cert\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.867308 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.867281 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-envoy\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.867597 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.867579 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-podinfo\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.872199 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.872180 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-token\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:04.872605 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:04.872584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctrb\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-kube-api-access-wctrb\") pod \"e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:05.017770 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.017741 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:05.293502 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.293478 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2"] Sep 17 02:00:05.294608 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:05.294584 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5c7f61_a2b1_422f_9483_e46b085c90a3.slice/crio-2e7d74c385fce1bec7b8dce9bb1183dbc482eb781829b8f2d58118ebc79225f0 WatchSource:0}: Error finding container 2e7d74c385fce1bec7b8dce9bb1183dbc482eb781829b8f2d58118ebc79225f0: Status 404 returned error can't find the container with id 2e7d74c385fce1bec7b8dce9bb1183dbc482eb781829b8f2d58118ebc79225f0 Sep 17 02:00:05.296964 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.296930 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:00:05.297068 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.297012 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:00:05.297068 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.297057 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:00:05.339915 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.339889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:05.440448 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.440416 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:05.744306 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.744269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" event={"ID":"6d5c7f61-a2b1-422f-9483-e46b085c90a3","Type":"ContainerStarted","Data":"6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995"} Sep 17 02:00:05.744739 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.744317 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" event={"ID":"6d5c7f61-a2b1-422f-9483-e46b085c90a3","Type":"ContainerStarted","Data":"2e7d74c385fce1bec7b8dce9bb1183dbc482eb781829b8f2d58118ebc79225f0"} Sep 17 02:00:05.768486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.768429 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podStartSLOduration=1.768408683 podStartE2EDuration="1.768408683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:05.762279688 +0000 UTC m=+1350.597332024" watchObservedRunningTime="2026-09-17 02:00:05.768408683 +0000 UTC m=+1350.603461018" Sep 17 02:00:05.875542 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.875520 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:05.883701 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.883674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:05.976172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.976129 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54b585e-343c-4e52-9c85-8c15141d7760-maas-api-service-ca\") pod \"c54b585e-343c-4e52-9c85-8c15141d7760\" (UID: \"c54b585e-343c-4e52-9c85-8c15141d7760\") " Sep 17 02:00:05.976497 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:05.976468 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54b585e-343c-4e52-9c85-8c15141d7760-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c54b585e-343c-4e52-9c85-8c15141d7760" (UID: "c54b585e-343c-4e52-9c85-8c15141d7760"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:06.018364 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.018293 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:06.023402 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.023373 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:06.077893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.077872 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54b585e-343c-4e52-9c85-8c15141d7760-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:06.614614 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.614577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:06.751241 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.751204 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" Sep 17 02:00:06.751241 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.751232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw" event={"ID":"c54b585e-343c-4e52-9c85-8c15141d7760","Type":"ContainerDied","Data":"83854e037251c5ea70667ab2067f8a50aac727428f4bb3372c1ea1f03d07aa10"} Sep 17 02:00:06.751645 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.751276 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83854e037251c5ea70667ab2067f8a50aac727428f4bb3372c1ea1f03d07aa10" Sep 17 02:00:06.751645 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.751615 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:06.752447 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:06.752429 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:08.012317 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:08.012277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:08.134106 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:08.134063 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:08.331178 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:08.331108 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:09.087865 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:09.087832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:09.901366 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:09.901324 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:10.586038 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:10.586000 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:11.330204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:11.330170 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:12.057200 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:12.057168 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:12.382970 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:12.382885 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:12.491584 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:12.491539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:12.883275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:12.883239 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:12.983658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:12.983617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:13.400167 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.400113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:13.502835 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.502793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:13.605088 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.605054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:13.656298 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.656201 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw"] Sep 17 02:00:13.661319 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.661264 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38758b-5q2vw"] Sep 17 02:00:13.681954 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.681928 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54b585e-343c-4e52-9c85-8c15141d7760" path="/var/lib/kubelet/pods/c54b585e-343c-4e52-9c85-8c15141d7760/volumes" Sep 17 02:00:13.685683 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:13.685656 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:14.039780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:14.039749 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:14.114696 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:14.114667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:15.051581 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:15.051549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:15.838019 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:15.837982 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:16.513542 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:16.513509 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:18.077480 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.077445 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" podUID="b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" containerName="main" containerID="cri-o://dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807" gracePeriod=45 Sep 17 02:00:18.120509 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.120479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:18.332284 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.332227 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 02:00:18.484272 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484244 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tmp-dir\") pod \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " Sep 17 02:00:18.484393 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484279 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-model-cache\") pod \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " Sep 17 02:00:18.484393 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484309 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-dshm\") pod \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " Sep 17 02:00:18.484393 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484350 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64lpv\" (UniqueName: \"kubernetes.io/projected/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-kube-api-access-64lpv\") pod \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " Sep 17 02:00:18.484393 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484383 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-home\") pod \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " Sep 17 02:00:18.484607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tls-certs\") pod \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\" (UID: \"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a\") " Sep 17 02:00:18.484607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484522 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" (UID: "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:18.484607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" (UID: "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:18.484766 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484701 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-home" (OuterVolumeSpecName: "home") pod "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" (UID: "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:18.484810 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484778 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:18.484810 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.484793 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:18.486649 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.486626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-dshm" (OuterVolumeSpecName: "dshm") pod "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" (UID: "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:18.486649 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.486631 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" (UID: "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:00:18.486803 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.486662 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-kube-api-access-64lpv" (OuterVolumeSpecName: "kube-api-access-64lpv") pod "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" (UID: "b53e2b5f-16cb-494b-af75-d6d0e1b9b18a"). InnerVolumeSpecName "kube-api-access-64lpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:18.585620 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.585569 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:18.585620 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.585590 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lpv\" (UniqueName: \"kubernetes.io/projected/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-kube-api-access-64lpv\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:18.585620 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.585599 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:18.585620 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.585608 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:18.817889 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.817864 2582 generic.go:356] "Generic (PLEG): container finished" podID="b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" containerID="dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807" exitCode=0 Sep 17 02:00:18.818017 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.817908 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" Sep 17 02:00:18.818017 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.817918 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" event={"ID":"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a","Type":"ContainerDied","Data":"dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807"} Sep 17 02:00:18.818017 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.817952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h" event={"ID":"b53e2b5f-16cb-494b-af75-d6d0e1b9b18a","Type":"ContainerDied","Data":"7ac26999bd6e37988a53ec19666c55d2da6acdc57b24bf44932cc2ca90d19a38"} Sep 17 02:00:18.818017 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.817968 2582 scope.go:117] "RemoveContainer" containerID="dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807" Sep 17 02:00:18.829868 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.829854 2582 scope.go:117] "RemoveContainer" containerID="dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807" Sep 17 02:00:18.830123 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:00:18.830109 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807\": container with ID starting with dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807 not found: ID does not exist" containerID="dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807" Sep 17 02:00:18.830172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.830134 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807"} err="failed to get container status \"dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807\": rpc error: code = NotFound desc = could not find container \"dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807\": container with ID starting with dacf7c352eb372d6525bc674dfe3ca623d87b7ccdf469627c851ac56214f1807 not found: ID does not exist" Sep 17 02:00:18.846655 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.846601 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h"] Sep 17 02:00:18.848583 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:18.848563 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-55c648/test-model-55c648-kserve-9d8869f67-5tt7h"] Sep 17 02:00:19.680336 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:19.680308 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" path="/var/lib/kubelet/pods/b53e2b5f-16cb-494b-af75-d6d0e1b9b18a/volumes" Sep 17 02:00:19.764616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:19.764585 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:19.823350 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:19.823326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:20.035261 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.035232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:20.378759 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.378682 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4"] Sep 17 02:00:20.379068 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.379056 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54b585e-343c-4e52-9c85-8c15141d7760" containerName="revoke-keys" Sep 17 02:00:20.379116 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.379071 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54b585e-343c-4e52-9c85-8c15141d7760" containerName="revoke-keys" Sep 17 02:00:20.379116 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.379090 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" containerName="main" Sep 17 02:00:20.379116 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.379096 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" containerName="main" Sep 17 02:00:20.379240 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.379154 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54b585e-343c-4e52-9c85-8c15141d7760" containerName="revoke-keys" Sep 17 02:00:20.379240 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.379165 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53e2b5f-16cb-494b-af75-d6d0e1b9b18a" containerName="main" Sep 17 02:00:20.383502 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.383485 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:20.386502 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.386479 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:00:20.386633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.386499 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:00:20.388247 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.388204 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4"] Sep 17 02:00:20.500250 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.500228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a568bf7-a7bd-418b-a4db-a99662a79f19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4\" (UID: \"1a568bf7-a7bd-418b-a4db-a99662a79f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:20.601470 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.601447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a568bf7-a7bd-418b-a4db-a99662a79f19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4\" (UID: \"1a568bf7-a7bd-418b-a4db-a99662a79f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:20.602047 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.602023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a568bf7-a7bd-418b-a4db-a99662a79f19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4\" (UID: \"1a568bf7-a7bd-418b-a4db-a99662a79f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:20.695974 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.695924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:20.965590 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:20.965565 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4"] Sep 17 02:00:20.967182 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:20.967151 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a568bf7_a7bd_418b_a4db_a99662a79f19.slice/crio-9fa28a672b5912b35fb184c10c56cec5c9ac7f496bedf194666bf8fe0b3db0b2 WatchSource:0}: Error finding container 9fa28a672b5912b35fb184c10c56cec5c9ac7f496bedf194666bf8fe0b3db0b2: Status 404 returned error can't find the container with id 9fa28a672b5912b35fb184c10c56cec5c9ac7f496bedf194666bf8fe0b3db0b2 Sep 17 02:00:21.135637 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:21.135613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:21.827998 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:21.827971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:21.835883 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:21.835861 2582 generic.go:356] "Generic (PLEG): container finished" podID="1a568bf7-a7bd-418b-a4db-a99662a79f19" containerID="82b99552d8d629631df07cef4203bf1bf446d7f86b954d86bd3853020b5a5044" exitCode=0 Sep 17 02:00:21.836028 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:21.836003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" event={"ID":"1a568bf7-a7bd-418b-a4db-a99662a79f19","Type":"ContainerDied","Data":"82b99552d8d629631df07cef4203bf1bf446d7f86b954d86bd3853020b5a5044"} Sep 17 02:00:21.836135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:21.836112 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" event={"ID":"1a568bf7-a7bd-418b-a4db-a99662a79f19","Type":"ContainerStarted","Data":"9fa28a672b5912b35fb184c10c56cec5c9ac7f496bedf194666bf8fe0b3db0b2"} Sep 17 02:00:22.544709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:22.544672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:22.972627 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:22.972605 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:23.122820 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.122758 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a568bf7-a7bd-418b-a4db-a99662a79f19-maas-api-service-ca\") pod \"1a568bf7-a7bd-418b-a4db-a99662a79f19\" (UID: \"1a568bf7-a7bd-418b-a4db-a99662a79f19\") " Sep 17 02:00:23.123088 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.123063 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a568bf7-a7bd-418b-a4db-a99662a79f19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a568bf7-a7bd-418b-a4db-a99662a79f19" (UID: "1a568bf7-a7bd-418b-a4db-a99662a79f19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:23.224251 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.224203 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a568bf7-a7bd-418b-a4db-a99662a79f19-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:23.305575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.305542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:23.847486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.847455 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" event={"ID":"1a568bf7-a7bd-418b-a4db-a99662a79f19","Type":"ContainerDied","Data":"9fa28a672b5912b35fb184c10c56cec5c9ac7f496bedf194666bf8fe0b3db0b2"} Sep 17 02:00:23.847624 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.847492 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4" Sep 17 02:00:23.847624 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:23.847504 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fa28a672b5912b35fb184c10c56cec5c9ac7f496bedf194666bf8fe0b3db0b2" Sep 17 02:00:24.282892 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:24.282855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:25.035655 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:25.035628 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:25.279165 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:25.279131 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:00:25.736063 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:25.736027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:26.328471 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:26.328426 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:27.149307 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:27.149272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:27.623022 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:27.622988 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:28.620967 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:28.620928 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:28.724118 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:28.724092 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:28.823807 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:28.823774 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:29.500928 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:29.500891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:30.532882 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:30.532839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.101773 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.101734 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.106132 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.106107 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g"] Sep 17 02:00:31.108874 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.108849 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-55c648-29826840-gd54g"] Sep 17 02:00:31.132370 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.132342 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w"] Sep 17 02:00:31.132808 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.132777 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" podUID="ea0e3b35-b7bc-4db8-834e-3b88f16f7665" containerName="payload-processing" containerID="cri-o://0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4" gracePeriod=30 Sep 17 02:00:31.202224 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.202177 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.289748 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.289700 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4"] Sep 17 02:00:31.292336 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.292304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.292587 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.292563 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55c648-zmnc4"] Sep 17 02:00:31.348916 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.348861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.447368 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.447339 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 02:00:31.497266 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.497238 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m475z\" (UniqueName: \"kubernetes.io/projected/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-kube-api-access-m475z\") pod \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " Sep 17 02:00:31.497429 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.497416 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-plugins-config-volume\") pod \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\" (UID: \"ea0e3b35-b7bc-4db8-834e-3b88f16f7665\") " Sep 17 02:00:31.497740 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.497717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea0e3b35-b7bc-4db8-834e-3b88f16f7665" (UID: "ea0e3b35-b7bc-4db8-834e-3b88f16f7665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:31.499206 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.499179 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-kube-api-access-m475z" (OuterVolumeSpecName: "kube-api-access-m475z") pod "ea0e3b35-b7bc-4db8-834e-3b88f16f7665" (UID: "ea0e3b35-b7bc-4db8-834e-3b88f16f7665"). InnerVolumeSpecName "kube-api-access-m475z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:31.598016 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.597989 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:31.598016 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.598014 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m475z\" (UniqueName: \"kubernetes.io/projected/ea0e3b35-b7bc-4db8-834e-3b88f16f7665-kube-api-access-m475z\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:31.679946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.679877 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a568bf7-a7bd-418b-a4db-a99662a79f19" path="/var/lib/kubelet/pods/1a568bf7-a7bd-418b-a4db-a99662a79f19/volumes" Sep 17 02:00:31.680187 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.680174 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7776039-a619-4752-89fe-78ceef3eff11" path="/var/lib/kubelet/pods/a7776039-a619-4752-89fe-78ceef3eff11/volumes" Sep 17 02:00:31.688484 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.688457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.787740 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.787701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:31.897355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.897324 2582 generic.go:356] "Generic (PLEG): container finished" podID="ea0e3b35-b7bc-4db8-834e-3b88f16f7665" containerID="0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4" exitCode=0 Sep 17 02:00:31.897527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.897368 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" Sep 17 02:00:31.897527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.897399 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" event={"ID":"ea0e3b35-b7bc-4db8-834e-3b88f16f7665","Type":"ContainerDied","Data":"0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4"} Sep 17 02:00:31.897527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.897451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w" event={"ID":"ea0e3b35-b7bc-4db8-834e-3b88f16f7665","Type":"ContainerDied","Data":"cfd30c68656b25a49bacb0a4abc353a45b2d7d3b2541f8d158ea4ad9b2ebe6ed"} Sep 17 02:00:31.897527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.897481 2582 scope.go:117] "RemoveContainer" containerID="0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4" Sep 17 02:00:31.906852 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.906829 2582 scope.go:117] "RemoveContainer" containerID="0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4" Sep 17 02:00:31.907113 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:00:31.907095 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4\": container with ID starting with 0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4 not found: ID does not exist" containerID="0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4" Sep 17 02:00:31.907169 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.907121 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4"} err="failed to get container status \"0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4\": rpc error: code = NotFound desc = could not find container \"0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4\": container with ID starting with 0b43f960a9a4d3f97939d8a2a46d8cc455338d96bfdf637f0060248b422aefd4 not found: ID does not exist" Sep 17 02:00:31.917156 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.917106 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w"] Sep 17 02:00:31.921331 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:31.921305 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c648-88d89d9d9-pr72w"] Sep 17 02:00:32.439930 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:32.439728 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:33.173780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:33.173742 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:33.686120 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:33.686081 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea0e3b35-b7bc-4db8-834e-3b88f16f7665" path="/var/lib/kubelet/pods/ea0e3b35-b7bc-4db8-834e-3b88f16f7665/volumes" Sep 17 02:00:34.086558 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:34.086524 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:34.951710 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:34.951680 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:35.008561 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.008535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:35.336418 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336385 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km"] Sep 17 02:00:35.336751 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336739 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a568bf7-a7bd-418b-a4db-a99662a79f19" containerName="revoke-keys" Sep 17 02:00:35.336792 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336753 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a568bf7-a7bd-418b-a4db-a99662a79f19" containerName="revoke-keys" Sep 17 02:00:35.336792 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336773 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0e3b35-b7bc-4db8-834e-3b88f16f7665" containerName="payload-processing" Sep 17 02:00:35.336792 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336779 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0e3b35-b7bc-4db8-834e-3b88f16f7665" containerName="payload-processing" Sep 17 02:00:35.336890 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336838 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a568bf7-a7bd-418b-a4db-a99662a79f19" containerName="revoke-keys" Sep 17 02:00:35.336890 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.336849 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0e3b35-b7bc-4db8-834e-3b88f16f7665" containerName="payload-processing" Sep 17 02:00:35.338806 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.338789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:35.341154 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.341136 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:00:35.341154 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.341145 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:00:35.347714 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.347680 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km"] Sep 17 02:00:35.435913 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.435887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km\" (UID: \"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:35.536480 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.536453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km\" (UID: \"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:35.538451 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.538434 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:00:35.547760 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.547731 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km\" (UID: \"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:35.653407 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.653340 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:00:35.661414 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.661396 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:35.943381 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:35.942532 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km"] Sep 17 02:00:35.944002 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:35.943973 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86fd37c_f6c1_4bda_8245_a8f2ec3f0696.slice/crio-2aab46841c3d899260d49e799bd8cae76786d154d1e092e932797b8622d799d4 WatchSource:0}: Error finding container 2aab46841c3d899260d49e799bd8cae76786d154d1e092e932797b8622d799d4: Status 404 returned error can't find the container with id 2aab46841c3d899260d49e799bd8cae76786d154d1e092e932797b8622d799d4 Sep 17 02:00:36.239721 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.239690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:36.753118 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.753087 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 17 02:00:36.753245 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.753137 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 17 02:00:36.914054 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.913979 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:36.930847 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.930819 2582 generic.go:356] "Generic (PLEG): container finished" podID="c86fd37c-f6c1-4bda-8245-a8f2ec3f0696" containerID="33d1fee540462bcbb019c55188101a465b8a1ede83767f47f9fb265e0a039d3d" exitCode=0 Sep 17 02:00:36.930958 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.930908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" event={"ID":"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696","Type":"ContainerDied","Data":"33d1fee540462bcbb019c55188101a465b8a1ede83767f47f9fb265e0a039d3d"} Sep 17 02:00:36.931012 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.930962 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" event={"ID":"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696","Type":"ContainerStarted","Data":"2aab46841c3d899260d49e799bd8cae76786d154d1e092e932797b8622d799d4"} Sep 17 02:00:36.932483 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.932464 2582 generic.go:356] "Generic (PLEG): container finished" podID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerID="6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995" exitCode=0 Sep 17 02:00:36.932575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.932507 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" event={"ID":"6d5c7f61-a2b1-422f-9483-e46b085c90a3","Type":"ContainerDied","Data":"6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995"} Sep 17 02:00:36.932848 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.932836 2582 scope.go:117] "RemoveContainer" containerID="6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995" Sep 17 02:00:36.933870 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.933842 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:00:36.933942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.933913 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:00:36.933981 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:36.933941 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:00:37.646482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:37.646446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:37.902117 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:37.901971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:37.942964 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:37.942921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" event={"ID":"6d5c7f61-a2b1-422f-9483-e46b085c90a3","Type":"ContainerStarted","Data":"68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c"} Sep 17 02:00:37.965754 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:37.965698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:38.043234 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.033567 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:38.043234 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.033697 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 17 02:00:38.043234 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.033738 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 17 02:00:38.097284 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.097264 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:38.160764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.160615 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696-maas-api-service-ca\") pod \"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696\" (UID: \"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696\") " Sep 17 02:00:38.160990 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.160957 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c86fd37c-f6c1-4bda-8245-a8f2ec3f0696" (UID: "c86fd37c-f6c1-4bda-8245-a8f2ec3f0696"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:00:38.161135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.161062 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:00:38.168929 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.168781 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:38.542220 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.542172 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:38.756200 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.756166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:38.949608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.949525 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" Sep 17 02:00:38.949608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.949561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km" event={"ID":"c86fd37c-f6c1-4bda-8245-a8f2ec3f0696","Type":"ContainerDied","Data":"2aab46841c3d899260d49e799bd8cae76786d154d1e092e932797b8622d799d4"} Sep 17 02:00:38.949608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:38.949609 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aab46841c3d899260d49e799bd8cae76786d154d1e092e932797b8622d799d4" Sep 17 02:00:39.019181 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:39.018767 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 17 02:00:39.019181 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:39.018817 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 17 02:00:39.074463 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:39.074426 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:39.175152 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:39.175116 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:40.018686 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:40.018650 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 17 02:00:40.019288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:40.019255 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 17 02:00:41.018938 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.018892 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 17 02:00:41.019449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.018968 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 17 02:00:41.662388 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.662353 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc"] Sep 17 02:00:41.662871 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.662854 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86fd37c-f6c1-4bda-8245-a8f2ec3f0696" containerName="revoke-keys" Sep 17 02:00:41.662922 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.662875 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86fd37c-f6c1-4bda-8245-a8f2ec3f0696" containerName="revoke-keys" Sep 17 02:00:41.662958 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.662954 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86fd37c-f6c1-4bda-8245-a8f2ec3f0696" containerName="revoke-keys" Sep 17 02:00:41.665156 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.665139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.667521 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.667498 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"distinct-2-w0-e564e1-kserve-self-signed-certs\"" Sep 17 02:00:41.680311 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.680289 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc"] Sep 17 02:00:41.694248 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.694225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tls-certs\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.694357 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.694276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptrq\" (UniqueName: \"kubernetes.io/projected/52fd9f12-461d-4f9f-ad9c-8cc073edd933-kube-api-access-8ptrq\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.694357 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.694297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-home\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.694447 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.694373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-dshm\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.694447 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.694420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-model-cache\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.694447 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.694438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tmp-dir\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.795833 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.795800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptrq\" (UniqueName: \"kubernetes.io/projected/52fd9f12-461d-4f9f-ad9c-8cc073edd933-kube-api-access-8ptrq\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.795833 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.795835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-home\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796047 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.795858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-dshm\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796047 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.795887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-model-cache\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796047 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.795905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tmp-dir\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796047 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.795963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tls-certs\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796276 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.796151 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-home\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796374 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.796354 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-model-cache\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.796435 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.796412 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tmp-dir\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.798038 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.798014 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-dshm\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.798438 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.798418 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tls-certs\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.806084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.806061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptrq\" (UniqueName: \"kubernetes.io/projected/52fd9f12-461d-4f9f-ad9c-8cc073edd933-kube-api-access-8ptrq\") pod \"distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:41.976041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:41.976014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:42.018777 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.018627 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:42.019117 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.019043 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 17 02:00:42.019117 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.019096 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 17 02:00:42.271625 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.271599 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc"] Sep 17 02:00:42.273407 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:42.273378 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52fd9f12_461d_4f9f_ad9c_8cc073edd933.slice/crio-951e5bd69d61ac2426220b215b310b1f31828e17399a422fa7f9e69a2770a998 WatchSource:0}: Error finding container 951e5bd69d61ac2426220b215b310b1f31828e17399a422fa7f9e69a2770a998: Status 404 returned error can't find the container with id 951e5bd69d61ac2426220b215b310b1f31828e17399a422fa7f9e69a2770a998 Sep 17 02:00:42.426808 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.426775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:42.528883 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.528790 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:42.971632 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.971603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" event={"ID":"52fd9f12-461d-4f9f-ad9c-8cc073edd933","Type":"ContainerStarted","Data":"2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601"} Sep 17 02:00:42.971804 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:42.971654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" event={"ID":"52fd9f12-461d-4f9f-ad9c-8cc073edd933","Type":"ContainerStarted","Data":"951e5bd69d61ac2426220b215b310b1f31828e17399a422fa7f9e69a2770a998"} Sep 17 02:00:43.022228 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:43.022195 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:43.022643 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:43.022630 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:43.023379 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:43.023357 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:00:43.041992 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:43.041952 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" podStartSLOduration=2.041938065 podStartE2EDuration="2.041938065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:42.989637833 +0000 UTC m=+1387.824690166" watchObservedRunningTime="2026-09-17 02:00:43.041938065 +0000 UTC m=+1387.876990397" Sep 17 02:00:43.269892 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:43.269818 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:44.150893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:44.150861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:45.057614 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.057585 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:45.156755 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.156723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:45.205424 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.205396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:45.520229 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.520178 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:45.620775 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.620748 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:45.664284 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.664261 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm"] Sep 17 02:00:45.666748 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.666731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.669025 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.669002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-67a23a\"" Sep 17 02:00:45.669141 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.669066 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-67a23a-dockercfg-d4lpw\"" Sep 17 02:00:45.681786 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.681760 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm"] Sep 17 02:00:45.734388 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.734362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjb8\" (UniqueName: \"kubernetes.io/projected/bcda171c-2cec-4968-88f4-3b4c725c7c4f-kube-api-access-dkjb8\") pod \"payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.734490 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.734416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcda171c-2cec-4968-88f4-3b4c725c7c4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.835240 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.835154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjb8\" (UniqueName: \"kubernetes.io/projected/bcda171c-2cec-4968-88f4-3b4c725c7c4f-kube-api-access-dkjb8\") pod \"payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.835240 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.835228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcda171c-2cec-4968-88f4-3b4c725c7c4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.835957 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.835932 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcda171c-2cec-4968-88f4-3b4c725c7c4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.842888 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.842848 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjb8\" (UniqueName: \"kubernetes.io/projected/bcda171c-2cec-4968-88f4-3b4c725c7c4f-kube-api-access-dkjb8\") pod \"payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:45.979283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:45.979258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:46.248677 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:46.248653 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm"] Sep 17 02:00:46.250465 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:00:46.250432 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcda171c_2cec_4968_88f4_3b4c725c7c4f.slice/crio-b837198bca6f717e070396d8a06c8d6f1227fa6d94c10d7b911df5878e3f87f1 WatchSource:0}: Error finding container b837198bca6f717e070396d8a06c8d6f1227fa6d94c10d7b911df5878e3f87f1: Status 404 returned error can't find the container with id b837198bca6f717e070396d8a06c8d6f1227fa6d94c10d7b911df5878e3f87f1 Sep 17 02:00:46.993862 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:46.993829 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" event={"ID":"bcda171c-2cec-4968-88f4-3b4c725c7c4f","Type":"ContainerStarted","Data":"f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41"} Sep 17 02:00:46.994029 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:46.993882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" event={"ID":"bcda171c-2cec-4968-88f4-3b4c725c7c4f","Type":"ContainerStarted","Data":"b837198bca6f717e070396d8a06c8d6f1227fa6d94c10d7b911df5878e3f87f1"} Sep 17 02:00:46.994029 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:46.993932 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:47.009627 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:47.009586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" podStartSLOduration=2.009573273 podStartE2EDuration="2.009573273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:47.007702377 +0000 UTC m=+1391.842754709" watchObservedRunningTime="2026-09-17 02:00:47.009573273 +0000 UTC m=+1391.844625605" Sep 17 02:00:49.062521 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.062484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:49.632473 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.632433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:49.733043 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.733009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:49.818467 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.818430 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km"] Sep 17 02:00:49.820850 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.820813 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:49.821343 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.821326 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a1f5690-lz2km"] Sep 17 02:00:49.887896 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:49.887832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:50.189977 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:50.189885 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:50.289407 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:50.289372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:51.111855 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:51.111818 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:51.680003 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:51.679970 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86fd37c-f6c1-4bda-8245-a8f2ec3f0696" path="/var/lib/kubelet/pods/c86fd37c-f6c1-4bda-8245-a8f2ec3f0696/volumes" Sep 17 02:00:51.873451 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:51.873425 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:51.976691 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:51.976667 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:51.976884 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:51.976705 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:51.988732 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:51.988711 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:52.032748 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:52.032729 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:00:52.734950 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:52.734914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:56.039079 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.039046 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:56.098973 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.098936 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:56.302538 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.302442 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2"] Sep 17 02:00:56.302731 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.302705 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" containerID="cri-o://68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c" gracePeriod=30 Sep 17 02:00:56.316059 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.316019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:56.688407 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.688316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:56.787309 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.787277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:56.920247 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:56.920192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:00:58.001050 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:58.001022 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:00:58.025715 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:58.025670 2582 patch_prober.go:28] interesting pod/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 02:00:58.025870 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:00:58.025752 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 02:01:01.553582 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.553559 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:01:01.679794 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679701 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-podinfo\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.679794 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679744 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istiod-ca-cert\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.679794 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679785 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-certs\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679818 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-envoy\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679914 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-token\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679947 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-credential-socket\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679975 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wctrb\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-kube-api-access-wctrb\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.679999 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-data\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680032 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-socket\") pod \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\" (UID: \"6d5c7f61-a2b1-422f-9483-e46b085c90a3\") " Sep 17 02:01:01.680431 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680188 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:01:01.680431 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680362 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.680431 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680368 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:01:01.680562 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680428 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:01:01.680606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680569 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:01:01.680760 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.680740 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:01:01.682247 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.682199 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:01:01.682369 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.682341 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:01:01.682433 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.682410 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 02:01:01.682549 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.682531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-kube-api-access-wctrb" (OuterVolumeSpecName: "kube-api-access-wctrb") pod "6d5c7f61-a2b1-422f-9483-e46b085c90a3" (UID: "6d5c7f61-a2b1-422f-9483-e46b085c90a3"). InnerVolumeSpecName "kube-api-access-wctrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:01:01.781536 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781497 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781536 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781530 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-credential-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781536 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781540 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wctrb\" (UniqueName: \"kubernetes.io/projected/6d5c7f61-a2b1-422f-9483-e46b085c90a3-kube-api-access-wctrb\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781791 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781549 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-data\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781791 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781557 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-workload-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781791 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781565 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-podinfo\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781791 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781574 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istiod-ca-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.781791 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.781581 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5c7f61-a2b1-422f-9483-e46b085c90a3-istio-envoy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:01:01.991520 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:01.991484 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:01:02.078654 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.078614 2582 generic.go:356] "Generic (PLEG): container finished" podID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerID="68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c" exitCode=0 Sep 17 02:01:02.078864 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.078663 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" Sep 17 02:01:02.078864 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.078702 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" event={"ID":"6d5c7f61-a2b1-422f-9483-e46b085c90a3","Type":"ContainerDied","Data":"68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c"} Sep 17 02:01:02.078864 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.078772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2" event={"ID":"6d5c7f61-a2b1-422f-9483-e46b085c90a3","Type":"ContainerDied","Data":"2e7d74c385fce1bec7b8dce9bb1183dbc482eb781829b8f2d58118ebc79225f0"} Sep 17 02:01:02.078864 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.078805 2582 scope.go:117] "RemoveContainer" containerID="68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c" Sep 17 02:01:02.089557 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.089533 2582 scope.go:117] "RemoveContainer" containerID="6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995" Sep 17 02:01:02.099734 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.099710 2582 scope.go:117] "RemoveContainer" containerID="68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c" Sep 17 02:01:02.100103 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:01:02.100081 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c\": container with ID starting with 68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c not found: ID does not exist" containerID="68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c" Sep 17 02:01:02.100169 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.100114 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c"} err="failed to get container status \"68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c\": rpc error: code = NotFound desc = could not find container \"68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c\": container with ID starting with 68bad22acf21c63b27c8671e8cd31401a242e7ee3e4576e25447a96ac69ea33c not found: ID does not exist" Sep 17 02:01:02.100169 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.100134 2582 scope.go:117] "RemoveContainer" containerID="6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995" Sep 17 02:01:02.100478 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:01:02.100457 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995\": container with ID starting with 6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995 not found: ID does not exist" containerID="6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995" Sep 17 02:01:02.100536 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.100483 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995"} err="failed to get container status \"6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995\": rpc error: code = NotFound desc = could not find container \"6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995\": container with ID starting with 6cda2b792421b48999b03ecdcbfb3a02055f1bb1fa9fb693ee65a7a99f75e995 not found: ID does not exist" Sep 17 02:01:02.103473 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.103439 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2"] Sep 17 02:01:02.107390 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.107356 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a1f5690-openshift-default-65b5f5d4cb-6g2q2"] Sep 17 02:01:02.818427 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:02.818390 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:03.412157 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:03.412119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:03.520600 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:03.520549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:03.612647 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:03.612612 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:03.679641 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:03.679557 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" path="/var/lib/kubelet/pods/6d5c7f61-a2b1-422f-9483-e46b085c90a3/volumes" Sep 17 02:01:03.795675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:03.795642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:03.895089 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:03.895054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:06.017458 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:06.017421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:07.375138 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:07.375103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:07.476417 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:07.476387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:07.727885 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:07.727845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:08.041475 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:08.041388 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:08.141479 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:08.141439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:08.244522 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:08.244468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:09.964607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:09.964571 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:10.717766 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:10.717736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:10.818177 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:10.818140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:10.941483 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:10.941453 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:11.246990 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.246949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:11.547975 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.547899 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth"] Sep 17 02:01:11.548419 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.548400 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" Sep 17 02:01:11.548515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.548420 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" Sep 17 02:01:11.548576 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.548524 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" Sep 17 02:01:11.551725 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.551702 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.554648 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.554621 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 02:01:11.555178 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.555157 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c5da3b-openshift-default-dockercfg-8hkcr\"" Sep 17 02:01:11.569387 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.569367 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth"] Sep 17 02:01:11.590768 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.590716 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:11.680633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-certs\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.680764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680636 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-podinfo\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.680764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-credential-socket\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.680764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-data\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.680764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-socket\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.680764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9fb5af-35ba-4549-9a10-62cdc030046a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.680764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-envoy\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.681085 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-token\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.681085 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.680843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4kp\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-kube-api-access-rm4kp\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.682715 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.682692 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:11.781715 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.781691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-envoy\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.781823 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.781736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-token\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.781823 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.781752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4kp\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-kube-api-access-rm4kp\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.781949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.781900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-certs\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.781949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.781935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-podinfo\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782069 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.781988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-credential-socket\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782069 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-data\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-socket\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9fb5af-35ba-4549-9a10-62cdc030046a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782341 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-certs\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782589 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782565 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-socket\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782682 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782570 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-credential-socket\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782682 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782663 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-data\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.782921 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.782904 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9fb5af-35ba-4549-9a10-62cdc030046a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.784519 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.784500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-envoy\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.784578 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.784559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-podinfo\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.789422 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.789399 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4kp\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-kube-api-access-rm4kp\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.789641 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.789620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-token\") pod \"e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.863198 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.863142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:11.931809 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:11.931763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:12.029816 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:12.029782 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:12.136945 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:12.136920 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth"] Sep 17 02:01:12.138849 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:01:12.138819 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9fb5af_35ba_4549_9a10_62cdc030046a.slice/crio-b27129862cfc473dcb9c41d2d8cf7f5d3a6d1322ab72d8d686661de6aba67a87 WatchSource:0}: Error finding container b27129862cfc473dcb9c41d2d8cf7f5d3a6d1322ab72d8d686661de6aba67a87: Status 404 returned error can't find the container with id b27129862cfc473dcb9c41d2d8cf7f5d3a6d1322ab72d8d686661de6aba67a87 Sep 17 02:01:12.140972 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:12.140936 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:12.141059 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:12.141013 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:12.141059 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:12.141042 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:12.565981 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:12.565945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:13.145989 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.145959 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" event={"ID":"8a9fb5af-35ba-4549-9a10-62cdc030046a","Type":"ContainerStarted","Data":"aa13f9b143f50668544d623c5e532c4b6ffb17e79576df08579b880a849ae02d"} Sep 17 02:01:13.146160 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.146008 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" event={"ID":"8a9fb5af-35ba-4549-9a10-62cdc030046a","Type":"ContainerStarted","Data":"b27129862cfc473dcb9c41d2d8cf7f5d3a6d1322ab72d8d686661de6aba67a87"} Sep 17 02:01:13.174103 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.174056 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podStartSLOduration=2.174041974 podStartE2EDuration="2.174041974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:13.172550304 +0000 UTC m=+1418.007602639" watchObservedRunningTime="2026-09-17 02:01:13.174041974 +0000 UTC m=+1418.009094306" Sep 17 02:01:13.259949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.259921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:13.358670 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.358643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:13.485295 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.485269 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:13.863838 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.863756 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:13.869502 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:13.869478 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:14.151784 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:14.151687 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:14.152588 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:14.152569 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:14.557082 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:14.557048 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:14.659926 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:14.659887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:14.900158 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:14.900061 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:15.060689 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.060646 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:15.160184 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.160094 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:15.243068 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.243026 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:15.244113 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.244090 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd"] Sep 17 02:01:15.244559 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.244541 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" Sep 17 02:01:15.244559 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.244560 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" Sep 17 02:01:15.244660 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.244647 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d5c7f61-a2b1-422f-9483-e46b085c90a3" containerName="istio-proxy" Sep 17 02:01:15.247121 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.247098 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.249722 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.249690 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f03337-dockercfg-zgnd5\"" Sep 17 02:01:15.249879 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.249861 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f03337\"" Sep 17 02:01:15.256684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.256652 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd"] Sep 17 02:01:15.316513 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.316470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.316751 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.316608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgff\" (UniqueName: \"kubernetes.io/projected/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-kube-api-access-9bgff\") pod \"payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.417671 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.417539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgff\" (UniqueName: \"kubernetes.io/projected/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-kube-api-access-9bgff\") pod \"payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.417671 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.417643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.418490 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.418465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.425953 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.425920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgff\" (UniqueName: \"kubernetes.io/projected/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-kube-api-access-9bgff\") pod \"payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.561043 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.560993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:15.577508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.577470 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:15.833974 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:15.833939 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd"] Sep 17 02:01:15.840362 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:01:15.840321 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7a724e_4ea9_43b9_87c0_bcc8433ef4e7.slice/crio-b1b8a8feb447fce51e066486c7bf6ffdb72927aa440810664ec556f0a04475b6 WatchSource:0}: Error finding container b1b8a8feb447fce51e066486c7bf6ffdb72927aa440810664ec556f0a04475b6: Status 404 returned error can't find the container with id b1b8a8feb447fce51e066486c7bf6ffdb72927aa440810664ec556f0a04475b6 Sep 17 02:01:16.165489 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:16.165384 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" event={"ID":"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7","Type":"ContainerStarted","Data":"2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7"} Sep 17 02:01:16.165489 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:16.165444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" event={"ID":"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7","Type":"ContainerStarted","Data":"b1b8a8feb447fce51e066486c7bf6ffdb72927aa440810664ec556f0a04475b6"} Sep 17 02:01:16.181714 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:16.181645 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" podStartSLOduration=1.181627397 podStartE2EDuration="1.181627397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:16.180973641 +0000 UTC m=+1421.016025975" watchObservedRunningTime="2026-09-17 02:01:16.181627397 +0000 UTC m=+1421.016679798" Sep 17 02:01:16.347315 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:16.347275 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:17.089191 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:17.089140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:17.171462 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:17.171420 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:17.934886 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:17.934847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:18.827160 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:18.827123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:19.453436 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:19.453398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:19.512399 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:19.512362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:19.930226 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:19.930175 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:20.034717 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.034659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:20.102165 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.102135 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh"] Sep 17 02:01:20.104953 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.104937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.107272 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.107256 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-c5da3b\"" Sep 17 02:01:20.107435 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.107416 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-c5da3b-dockercfg-shj92\"" Sep 17 02:01:20.115939 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.115917 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh"] Sep 17 02:01:20.160604 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.160578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj67t\" (UniqueName: \"kubernetes.io/projected/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-kube-api-access-mj67t\") pod \"payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.160777 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.160751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.261578 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.261549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.261698 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.261626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj67t\" (UniqueName: \"kubernetes.io/projected/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-kube-api-access-mj67t\") pod \"payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.262303 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.262287 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.268948 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.268928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj67t\" (UniqueName: \"kubernetes.io/projected/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-kube-api-access-mj67t\") pod \"payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.416249 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.416218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:20.686863 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.686832 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh"] Sep 17 02:01:20.687962 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:01:20.687936 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f805c03_4256_42c9_81c5_f83dd6ba4bbf.slice/crio-03990abc7d7d2b982000048ea6532a2cd416f92c205d1b6d522b84190fb3b962 WatchSource:0}: Error finding container 03990abc7d7d2b982000048ea6532a2cd416f92c205d1b6d522b84190fb3b962: Status 404 returned error can't find the container with id 03990abc7d7d2b982000048ea6532a2cd416f92c205d1b6d522b84190fb3b962 Sep 17 02:01:20.756357 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:20.756329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:21.193830 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:21.193798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" event={"ID":"9f805c03-4256-42c9-81c5-f83dd6ba4bbf","Type":"ContainerStarted","Data":"c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207"} Sep 17 02:01:21.194231 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:21.193850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" event={"ID":"9f805c03-4256-42c9-81c5-f83dd6ba4bbf","Type":"ContainerStarted","Data":"03990abc7d7d2b982000048ea6532a2cd416f92c205d1b6d522b84190fb3b962"} Sep 17 02:01:21.194231 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:21.193872 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:21.208783 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:21.208737 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" podStartSLOduration=1.20872148 podStartE2EDuration="1.20872148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:21.207397205 +0000 UTC m=+1426.042449537" watchObservedRunningTime="2026-09-17 02:01:21.20872148 +0000 UTC m=+1426.043774108" Sep 17 02:01:24.712085 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:24.712053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:24.778756 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:24.778726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:28.178146 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:28.178065 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:01:29.575875 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:29.575842 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:01:32.201385 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:32.201351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:01:35.562750 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:35.562718 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:36.572558 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:36.572491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:37.062290 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:37.062242 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:37.776225 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:37.776163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:38.349677 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:38.349634 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:39.150840 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:39.150804 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:39.817505 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:39.817469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:40.308838 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.308814 2582 generic.go:356] "Generic (PLEG): container finished" podID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerID="aa13f9b143f50668544d623c5e532c4b6ffb17e79576df08579b880a849ae02d" exitCode=0 Sep 17 02:01:40.309171 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.308886 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" event={"ID":"8a9fb5af-35ba-4549-9a10-62cdc030046a","Type":"ContainerDied","Data":"aa13f9b143f50668544d623c5e532c4b6ffb17e79576df08579b880a849ae02d"} Sep 17 02:01:40.309334 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.309320 2582 scope.go:117] "RemoveContainer" containerID="aa13f9b143f50668544d623c5e532c4b6ffb17e79576df08579b880a849ae02d" Sep 17 02:01:40.310466 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.310439 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:40.310527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.310505 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:40.310563 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.310536 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:40.839863 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:40.839826 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:41.327136 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:41.327089 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" event={"ID":"8a9fb5af-35ba-4549-9a10-62cdc030046a","Type":"ContainerStarted","Data":"e0a13f6917a514c597635545c4f1ef59a6bacd34005207718097eec4ca86586b"} Sep 17 02:01:41.863413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:41.863378 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:41.864289 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:41.864265 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 17 02:01:41.864378 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:41.864332 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 17 02:01:42.864155 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:42.864116 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 17 02:01:42.864596 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:42.864183 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 17 02:01:43.808957 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:43.808923 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:43.864104 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:43.864066 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 17 02:01:43.864305 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:43.864141 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 17 02:01:44.151849 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:44.151754 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:44.699456 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:44.699406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:44.864330 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:44.864290 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 17 02:01:44.864760 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:44.864366 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 17 02:01:45.310807 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.310771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:45.678125 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.678021 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:45.776447 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.776406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:45.863589 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.863552 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 17 02:01:45.863774 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.863613 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 17 02:01:45.989694 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.989653 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5"] Sep 17 02:01:45.992425 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.992405 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:45.994765 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:45.994741 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a3912d-openshift-default-dockercfg-2zl2k\"" Sep 17 02:01:46.008312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.008281 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5"] Sep 17 02:01:46.121868 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.121825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-token\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.122166 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.122146 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-envoy\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.122389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.122369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-data\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.122551 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.122534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-podinfo\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.122680 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.122664 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788gk\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-kube-api-access-788gk\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.122818 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.122803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-certs\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.122960 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.122945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-credential-socket\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.123091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.123077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-socket\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.123270 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.123240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istiod-ca-cert\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224090 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istiod-ca-cert\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224325 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-token\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224325 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-envoy\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224325 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224223 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-data\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224325 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-podinfo\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224325 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224306 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788gk\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-kube-api-access-788gk\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224617 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-certs\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224617 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224409 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-credential-socket\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224617 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-socket\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-socket\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.224894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.224879 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istiod-ca-cert\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.225656 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.225628 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-credential-socket\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.225832 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.225812 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-certs\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.225946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.225924 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-data\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.228039 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.228013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-podinfo\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.228166 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.228104 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-envoy\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.236603 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.236577 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-token\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.238557 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.238534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788gk\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-kube-api-access-788gk\") pod \"e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.306250 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.306118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:46.527905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.527871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:46.606242 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.606218 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5"] Sep 17 02:01:46.608259 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:01:46.608227 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0bff41_99a0_46c3_9b4f_7b17f1ce2478.slice/crio-1f8b9be145c935cad5b86bc02c6be330bda710fa066101f23ceac1dd62959601 WatchSource:0}: Error finding container 1f8b9be145c935cad5b86bc02c6be330bda710fa066101f23ceac1dd62959601: Status 404 returned error can't find the container with id 1f8b9be145c935cad5b86bc02c6be330bda710fa066101f23ceac1dd62959601 Sep 17 02:01:46.608259 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.608240 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:46.610595 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.610568 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:46.610674 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.610628 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:46.610674 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.610656 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:01:46.867891 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.867812 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:46.868910 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:46.868884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:01:47.377289 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:47.377229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" event={"ID":"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478","Type":"ContainerStarted","Data":"cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d"} Sep 17 02:01:47.377786 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:47.377314 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" event={"ID":"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478","Type":"ContainerStarted","Data":"1f8b9be145c935cad5b86bc02c6be330bda710fa066101f23ceac1dd62959601"} Sep 17 02:01:47.399245 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:47.399161 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podStartSLOduration=2.399126186 podStartE2EDuration="2.399126186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:47.395994104 +0000 UTC m=+1452.231046437" watchObservedRunningTime="2026-09-17 02:01:47.399126186 +0000 UTC m=+1452.234178519" Sep 17 02:01:48.306533 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:48.306494 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:48.312441 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:48.312413 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:48.381508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:48.381469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:48.382633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:48.382611 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:01:48.611417 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:48.611317 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:49.162256 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:49.162196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:49.314407 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:49.314370 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:49.408038 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:49.408005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:50.087560 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:50.087520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:50.753403 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:50.753374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:51.522787 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:51.522751 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:52.176827 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:52.176797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:52.854806 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:52.854775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:52.956643 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:52.956610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:53.394631 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.394583 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:01:53.399883 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.399854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:53.503591 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.503558 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:01:53.530057 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.530028 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f"] Sep 17 02:01:53.533268 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.533252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.537282 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.537254 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a3912d\"" Sep 17 02:01:53.542781 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.542760 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5"] Sep 17 02:01:53.545639 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.545620 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.546880 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.546862 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f"] Sep 17 02:01:53.548120 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.548100 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a3912d-dockercfg-fmw7b\"" Sep 17 02:01:53.558702 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.558684 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5"] Sep 17 02:01:53.591592 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.591571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgf8x\" (UniqueName: \"kubernetes.io/projected/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-kube-api-access-dgf8x\") pod \"payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.591694 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.591617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2fq\" (UniqueName: \"kubernetes.io/projected/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-kube-api-access-kr2fq\") pod \"payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.591743 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.591692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.591780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.591745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.692320 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.692245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.692468 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.692323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.692468 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.692386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgf8x\" (UniqueName: \"kubernetes.io/projected/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-kube-api-access-dgf8x\") pod \"payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.692578 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.692509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2fq\" (UniqueName: \"kubernetes.io/projected/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-kube-api-access-kr2fq\") pod \"payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.693024 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.693005 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.693114 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.693096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.701478 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.701448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2fq\" (UniqueName: \"kubernetes.io/projected/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-kube-api-access-kr2fq\") pod \"payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.701573 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.701448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgf8x\" (UniqueName: \"kubernetes.io/projected/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-kube-api-access-dgf8x\") pod \"payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:53.845041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.845004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:53.856316 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:53.856298 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:54.120280 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.120253 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f"] Sep 17 02:01:54.121570 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:01:54.121541 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9a3fcc_90ee_4880_93e9_b58fd61c882b.slice/crio-4db8ab3b61387d0c222aa92664b8a19121670c464cdd738cac62859c8168fb87 WatchSource:0}: Error finding container 4db8ab3b61387d0c222aa92664b8a19121670c464cdd738cac62859c8168fb87: Status 404 returned error can't find the container with id 4db8ab3b61387d0c222aa92664b8a19121670c464cdd738cac62859c8168fb87 Sep 17 02:01:54.139295 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:01:54.139272 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9a6bf2_1df9_4372_9f9b_d26d5a30eb54.slice/crio-9b641f002fd3db3071ea403d53fcaa7a19c4e9202e86f6a4372887f495258c25 WatchSource:0}: Error finding container 9b641f002fd3db3071ea403d53fcaa7a19c4e9202e86f6a4372887f495258c25: Status 404 returned error can't find the container with id 9b641f002fd3db3071ea403d53fcaa7a19c4e9202e86f6a4372887f495258c25 Sep 17 02:01:54.139625 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.139605 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5"] Sep 17 02:01:54.424300 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.424186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" event={"ID":"9c9a3fcc-90ee-4880-93e9-b58fd61c882b","Type":"ContainerStarted","Data":"6d4ac353527f3c736ed5fce7307262f36f7932cdf44ef59c340ab8127b82f874"} Sep 17 02:01:54.424744 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.424314 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" event={"ID":"9c9a3fcc-90ee-4880-93e9-b58fd61c882b","Type":"ContainerStarted","Data":"4db8ab3b61387d0c222aa92664b8a19121670c464cdd738cac62859c8168fb87"} Sep 17 02:01:54.424959 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.424936 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:01:54.426968 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.426936 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" event={"ID":"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54","Type":"ContainerStarted","Data":"1101018e6f6275c8928db76a0e92522d74c4bf33e3f990424a68fb6ead9ebfe8"} Sep 17 02:01:54.427103 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.426986 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" event={"ID":"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54","Type":"ContainerStarted","Data":"9b641f002fd3db3071ea403d53fcaa7a19c4e9202e86f6a4372887f495258c25"} Sep 17 02:01:54.427103 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.427082 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:01:54.441755 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.441713 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" podStartSLOduration=1.441701494 podStartE2EDuration="1.441701494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:54.44011387 +0000 UTC m=+1459.275166207" watchObservedRunningTime="2026-09-17 02:01:54.441701494 +0000 UTC m=+1459.276753827" Sep 17 02:01:54.458252 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:54.458197 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" podStartSLOduration=1.458185872 podStartE2EDuration="1.458185872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:54.455719394 +0000 UTC m=+1459.290771729" watchObservedRunningTime="2026-09-17 02:01:54.458185872 +0000 UTC m=+1459.293238202" Sep 17 02:01:56.873839 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:01:56.873803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:05.434888 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:05.434857 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:02:05.435267 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:05.434930 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:02:15.158460 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:15.158419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:15.853264 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:15.853205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:16.651089 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:16.651050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:17.241232 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:17.240324 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:18.034841 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:18.034801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:18.831192 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:18.831152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:19.423361 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:19.423325 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 02:02:19.423741 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:19.423387 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 02:02:19.485200 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:19.485162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:19.640391 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:19.640360 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:21.590760 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:21.590730 2582 generic.go:356] "Generic (PLEG): container finished" podID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerID="cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d" exitCode=0 Sep 17 02:02:21.591162 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:21.590768 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" event={"ID":"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478","Type":"ContainerDied","Data":"cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d"} Sep 17 02:02:21.591162 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:21.591138 2582 scope.go:117] "RemoveContainer" containerID="cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d" Sep 17 02:02:21.592375 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:21.592343 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:02:21.592449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:21.592414 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:02:21.592488 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:21.592446 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 02:02:22.598254 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:22.598198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" event={"ID":"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478","Type":"ContainerStarted","Data":"292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1"} Sep 17 02:02:23.306658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:23.306621 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:02:23.306824 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:23.306793 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 17 02:02:23.306869 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:23.306836 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 17 02:02:24.306608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:24.306575 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 17 02:02:24.306971 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:24.306648 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 17 02:02:25.307038 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:25.307006 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 17 02:02:25.307437 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:25.307090 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 17 02:02:26.307257 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.307183 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 17 02:02:26.307708 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.307279 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 17 02:02:26.383607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.383570 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d"] Sep 17 02:02:26.386355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.386336 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.389996 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.389969 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"auth-test-model-f03337-kserve-self-signed-certs\"" Sep 17 02:02:26.391223 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.391185 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"kube-root-ca.crt\"" Sep 17 02:02:26.391355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.391265 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"openshift-service-ca.crt\"" Sep 17 02:02:26.391355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.391284 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"default-dockercfg-xb9xt\"" Sep 17 02:02:26.410716 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.410681 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d"] Sep 17 02:02:26.445133 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.445096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:26.502494 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.502452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700cdeee-8acf-4668-96ff-61c7c16fb10d-tls-certs\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.502716 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.502533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-home\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.502716 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.502585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfrq\" (UniqueName: \"kubernetes.io/projected/700cdeee-8acf-4668-96ff-61c7c16fb10d-kube-api-access-dtfrq\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.502716 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.502692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-model-cache\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.502861 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.502731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-tmp-dir\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.502861 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.502805 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-dshm\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.603800 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.603685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-home\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.603800 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.603748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfrq\" (UniqueName: \"kubernetes.io/projected/700cdeee-8acf-4668-96ff-61c7c16fb10d-kube-api-access-dtfrq\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604042 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.603818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-model-cache\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604042 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.603846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-tmp-dir\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604042 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.603891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-dshm\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604042 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.603926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700cdeee-8acf-4668-96ff-61c7c16fb10d-tls-certs\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604336 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.604097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-home\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604336 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.604271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-model-cache\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.604336 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.604304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-tmp-dir\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.606225 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.606191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-dshm\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.606644 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.606623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700cdeee-8acf-4668-96ff-61c7c16fb10d-tls-certs\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.612748 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.612718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfrq\" (UniqueName: \"kubernetes.io/projected/700cdeee-8acf-4668-96ff-61c7c16fb10d-kube-api-access-dtfrq\") pod \"auth-test-model-f03337-kserve-cb7ffbc55-jlk9d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.696758 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.696722 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:26.995952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:26.995853 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d"] Sep 17 02:02:26.998620 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:02:26.998572 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod700cdeee_8acf_4668_96ff_61c7c16fb10d.slice/crio-e6b2fdaa4818db3ac50fa32fbd675843cc1025bdeda5603b70acfbb6aa9599d0 WatchSource:0}: Error finding container e6b2fdaa4818db3ac50fa32fbd675843cc1025bdeda5603b70acfbb6aa9599d0: Status 404 returned error can't find the container with id e6b2fdaa4818db3ac50fa32fbd675843cc1025bdeda5603b70acfbb6aa9599d0 Sep 17 02:02:27.307179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:27.307099 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 17 02:02:27.307179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:27.307146 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 17 02:02:27.654481 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:27.654382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" event={"ID":"700cdeee-8acf-4668-96ff-61c7c16fb10d","Type":"ContainerStarted","Data":"089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86"} Sep 17 02:02:27.654481 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:27.654439 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" event={"ID":"700cdeee-8acf-4668-96ff-61c7c16fb10d","Type":"ContainerStarted","Data":"e6b2fdaa4818db3ac50fa32fbd675843cc1025bdeda5603b70acfbb6aa9599d0"} Sep 17 02:02:27.674725 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:27.674670 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" podStartSLOduration=1.6746554009999999 podStartE2EDuration="1.674655401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:27.671939275 +0000 UTC m=+1492.506991608" watchObservedRunningTime="2026-09-17 02:02:27.674655401 +0000 UTC m=+1492.509707734" Sep 17 02:02:28.310718 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:28.310685 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:02:28.311170 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:28.310972 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:02:28.311629 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:28.311609 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:02:29.467024 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:29.466990 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:02:33.181370 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:33.181332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:33.552656 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:33.552611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:34.300634 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:34.300595 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:35.751614 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:35.751583 2582 scope.go:117] "RemoveContainer" containerID="b481eb22a224ff66d2d915fe55c4523d00cea2121d4461e8895876e2990a4a9e" Sep 17 02:02:36.334607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:36.334571 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:36.697442 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:36.697361 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:36.697442 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:36.697404 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:36.710079 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:36.710052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:36.724680 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:36.724651 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:02:36.853531 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:36.853501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:37.050108 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:37.050078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:37.150612 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:37.150577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:37.636272 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:37.636235 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:38.497450 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:38.497391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:39.148791 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:39.148756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:39.805398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:39.805361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:40.646546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:40.646512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:41.337628 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:41.337598 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:41.436060 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:41.436022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:41.723449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:41.723421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:41.839936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:41.839902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:02:41.913868 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:02:41.913838 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:03.767829 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:03.767791 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:04.588111 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:04.588069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:05.566343 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:05.566308 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:06.372932 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:06.372896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:06.937225 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:06.937181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:07.195897 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:07.195813 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:07.741164 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:07.741121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:08.550506 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:08.550470 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:09.433966 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:09.433933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:13.800531 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:13.800488 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:03:13.831579 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:13.831537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:35.771486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:35.771452 2582 scope.go:117] "RemoveContainer" containerID="a0aaf07243aa2db67692b47789eeefde3c09b9a1e4c8f127e9129ed99887dd50" Sep 17 02:03:35.780014 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:35.779996 2582 scope.go:117] "RemoveContainer" containerID="4ce18cf6113861041e38132d12b2f3291220473062065099dc9fa293a2c0d4f4" Sep 17 02:03:48.969914 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:48.969882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:03:55.547559 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.547517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:55.661758 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.661724 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4"] Sep 17 02:03:55.664395 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.664364 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:55.666608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.666590 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:03:55.666608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.666599 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:03:55.672413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.672392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4"] Sep 17 02:03:55.831350 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.831280 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff85406-d413-476b-9309-19ec6483e2e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4\" (UID: \"bff85406-d413-476b-9309-19ec6483e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:55.932156 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.932133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff85406-d413-476b-9309-19ec6483e2e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4\" (UID: \"bff85406-d413-476b-9309-19ec6483e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:55.932718 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.932699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff85406-d413-476b-9309-19ec6483e2e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4\" (UID: \"bff85406-d413-476b-9309-19ec6483e2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:55.943049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.943026 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:55.976000 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:55.975980 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:56.250353 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:56.250329 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4"] Sep 17 02:03:56.252178 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:03:56.252140 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff85406_d413_476b_9309_19ec6483e2e5.slice/crio-08709fdd536115fbe2a0008e83831c1c0ca9c9fa8bfbb6a274abf0bdc23b08c0 WatchSource:0}: Error finding container 08709fdd536115fbe2a0008e83831c1c0ca9c9fa8bfbb6a274abf0bdc23b08c0: Status 404 returned error can't find the container with id 08709fdd536115fbe2a0008e83831c1c0ca9c9fa8bfbb6a274abf0bdc23b08c0 Sep 17 02:03:56.659151 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:56.659067 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:57.130679 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.130652 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg"] Sep 17 02:03:57.133484 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.133466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:03:57.142219 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.142192 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg"] Sep 17 02:03:57.169511 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.169446 2582 generic.go:356] "Generic (PLEG): container finished" podID="bff85406-d413-476b-9309-19ec6483e2e5" containerID="9ab8423b37533f76b341ceca664113a0ce33bb6776b4661d59338cf03c43207e" exitCode=0 Sep 17 02:03:57.169616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.169508 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" event={"ID":"bff85406-d413-476b-9309-19ec6483e2e5","Type":"ContainerDied","Data":"9ab8423b37533f76b341ceca664113a0ce33bb6776b4661d59338cf03c43207e"} Sep 17 02:03:57.169616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.169544 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" event={"ID":"bff85406-d413-476b-9309-19ec6483e2e5","Type":"ContainerStarted","Data":"08709fdd536115fbe2a0008e83831c1c0ca9c9fa8bfbb6a274abf0bdc23b08c0"} Sep 17 02:03:57.246804 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.246781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54f8810-216e-48fd-bfcf-c6bb7870bb60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg\" (UID: \"b54f8810-216e-48fd-bfcf-c6bb7870bb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:03:57.348161 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.348133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54f8810-216e-48fd-bfcf-c6bb7870bb60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg\" (UID: \"b54f8810-216e-48fd-bfcf-c6bb7870bb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:03:57.348747 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.348731 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54f8810-216e-48fd-bfcf-c6bb7870bb60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg\" (UID: \"b54f8810-216e-48fd-bfcf-c6bb7870bb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:03:57.445119 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.445067 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:03:57.607871 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.607833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:57.712796 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:57.712771 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg"] Sep 17 02:03:57.714567 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:03:57.714531 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54f8810_216e_48fd_bfcf_c6bb7870bb60.slice/crio-074fa18ab0ff2471031a5e3ecbd45b26b2b17ed2f829ee54aea43791a9d2448d WatchSource:0}: Error finding container 074fa18ab0ff2471031a5e3ecbd45b26b2b17ed2f829ee54aea43791a9d2448d: Status 404 returned error can't find the container with id 074fa18ab0ff2471031a5e3ecbd45b26b2b17ed2f829ee54aea43791a9d2448d Sep 17 02:03:58.176902 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.176831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" event={"ID":"b54f8810-216e-48fd-bfcf-c6bb7870bb60","Type":"ContainerStarted","Data":"1489734c970f9ded7bee52e31daa460a1c8f5f21d8e49673c29fda1f8b196785"} Sep 17 02:03:58.176902 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.176883 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" event={"ID":"b54f8810-216e-48fd-bfcf-c6bb7870bb60","Type":"ContainerStarted","Data":"074fa18ab0ff2471031a5e3ecbd45b26b2b17ed2f829ee54aea43791a9d2448d"} Sep 17 02:03:58.190686 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.190637 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" podStartSLOduration=1.190620049 podStartE2EDuration="1.190620049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:03:58.18988122 +0000 UTC m=+1583.024933553" watchObservedRunningTime="2026-09-17 02:03:58.190620049 +0000 UTC m=+1583.025672383" Sep 17 02:03:58.305301 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.305274 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:58.458764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.458704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff85406-d413-476b-9309-19ec6483e2e5-maas-api-service-ca\") pod \"bff85406-d413-476b-9309-19ec6483e2e5\" (UID: \"bff85406-d413-476b-9309-19ec6483e2e5\") " Sep 17 02:03:58.459027 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.459006 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff85406-d413-476b-9309-19ec6483e2e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bff85406-d413-476b-9309-19ec6483e2e5" (UID: "bff85406-d413-476b-9309-19ec6483e2e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:03:58.536639 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.536613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:58.559843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:58.559818 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bff85406-d413-476b-9309-19ec6483e2e5-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:03:59.178429 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:59.178396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:03:59.188658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:59.188636 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" Sep 17 02:03:59.188805 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:59.188671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4" event={"ID":"bff85406-d413-476b-9309-19ec6483e2e5","Type":"ContainerDied","Data":"08709fdd536115fbe2a0008e83831c1c0ca9c9fa8bfbb6a274abf0bdc23b08c0"} Sep 17 02:03:59.188805 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:59.188749 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08709fdd536115fbe2a0008e83831c1c0ca9c9fa8bfbb6a274abf0bdc23b08c0" Sep 17 02:03:59.190405 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:59.190376 2582 generic.go:356] "Generic (PLEG): container finished" podID="b54f8810-216e-48fd-bfcf-c6bb7870bb60" containerID="1489734c970f9ded7bee52e31daa460a1c8f5f21d8e49673c29fda1f8b196785" exitCode=0 Sep 17 02:03:59.190530 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:03:59.190443 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" event={"ID":"b54f8810-216e-48fd-bfcf-c6bb7870bb60","Type":"ContainerDied","Data":"1489734c970f9ded7bee52e31daa460a1c8f5f21d8e49673c29fda1f8b196785"} Sep 17 02:04:00.023986 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.023953 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:00.340191 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.340168 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:04:00.478323 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.478293 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54f8810-216e-48fd-bfcf-c6bb7870bb60-maas-api-service-ca\") pod \"b54f8810-216e-48fd-bfcf-c6bb7870bb60\" (UID: \"b54f8810-216e-48fd-bfcf-c6bb7870bb60\") " Sep 17 02:04:00.478670 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.478651 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54f8810-216e-48fd-bfcf-c6bb7870bb60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b54f8810-216e-48fd-bfcf-c6bb7870bb60" (UID: "b54f8810-216e-48fd-bfcf-c6bb7870bb60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:04:00.548128 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.548098 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp"] Sep 17 02:04:00.556991 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.556967 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66"] Sep 17 02:04:00.579859 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.579805 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b54f8810-216e-48fd-bfcf-c6bb7870bb60-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:00.620983 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.620958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:00.722073 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:00.722038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:01.202752 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:01.202724 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" Sep 17 02:04:01.202752 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:01.202742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg" event={"ID":"b54f8810-216e-48fd-bfcf-c6bb7870bb60","Type":"ContainerDied","Data":"074fa18ab0ff2471031a5e3ecbd45b26b2b17ed2f829ee54aea43791a9d2448d"} Sep 17 02:04:01.202992 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:01.202785 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074fa18ab0ff2471031a5e3ecbd45b26b2b17ed2f829ee54aea43791a9d2448d" Sep 17 02:04:01.304410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:01.304374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:02.063382 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:02.063350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:02.746965 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:02.746933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:03.557894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:03.557859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:04.573547 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.573520 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd"] Sep 17 02:04:04.573936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.573921 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b54f8810-216e-48fd-bfcf-c6bb7870bb60" containerName="revoke-keys" Sep 17 02:04:04.573985 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.573938 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b54f8810-216e-48fd-bfcf-c6bb7870bb60" containerName="revoke-keys" Sep 17 02:04:04.573985 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.573963 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff85406-d413-476b-9309-19ec6483e2e5" containerName="revoke-keys" Sep 17 02:04:04.573985 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.573969 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff85406-d413-476b-9309-19ec6483e2e5" containerName="revoke-keys" Sep 17 02:04:04.574142 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.574031 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b54f8810-216e-48fd-bfcf-c6bb7870bb60" containerName="revoke-keys" Sep 17 02:04:04.574142 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.574043 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff85406-d413-476b-9309-19ec6483e2e5" containerName="revoke-keys" Sep 17 02:04:04.576175 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.576159 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.578412 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.578393 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e564e1\"/\"embedding-w0-e564e1-kserve-self-signed-certs\"" Sep 17 02:04:04.586504 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.586480 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd"] Sep 17 02:04:04.675862 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.675835 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:04.718555 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.718531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-tmp-dir\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.718674 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.718571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-model-cache\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.718674 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.718634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8069c44-7652-42a2-a27f-bf7670206814-tls-certs\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.718674 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.718657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-dshm\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.718837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.718777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgwf\" (UniqueName: \"kubernetes.io/projected/e8069c44-7652-42a2-a27f-bf7670206814-kube-api-access-rxgwf\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.718837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.718828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-home\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819412 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-tmp-dir\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819543 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-model-cache\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819543 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8069c44-7652-42a2-a27f-bf7670206814-tls-certs\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819543 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-dshm\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819717 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgwf\" (UniqueName: \"kubernetes.io/projected/e8069c44-7652-42a2-a27f-bf7670206814-kube-api-access-rxgwf\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819717 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-home\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819830 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819813 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-tmp-dir\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.819886 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.819840 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-model-cache\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.820025 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.820003 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-home\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.821769 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.821744 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-dshm\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.822018 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.822000 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8069c44-7652-42a2-a27f-bf7670206814-tls-certs\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.826927 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.826877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgwf\" (UniqueName: \"kubernetes.io/projected/e8069c44-7652-42a2-a27f-bf7670206814-kube-api-access-rxgwf\") pod \"embedding-w0-e564e1-kserve-6444745b58-vflwd\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:04.886078 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:04.886047 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:05.158310 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:05.158281 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd"] Sep 17 02:04:05.159707 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:04:05.159685 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8069c44_7652_42a2_a27f_bf7670206814.slice/crio-4238e2f4950e48b62749d8f1723cc156b01165ed4c2af0108b851a079d78ee97 WatchSource:0}: Error finding container 4238e2f4950e48b62749d8f1723cc156b01165ed4c2af0108b851a079d78ee97: Status 404 returned error can't find the container with id 4238e2f4950e48b62749d8f1723cc156b01165ed4c2af0108b851a079d78ee97 Sep 17 02:04:05.226265 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:05.226236 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" event={"ID":"e8069c44-7652-42a2-a27f-bf7670206814","Type":"ContainerStarted","Data":"80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93"} Sep 17 02:04:05.226356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:05.226283 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" event={"ID":"e8069c44-7652-42a2-a27f-bf7670206814","Type":"ContainerStarted","Data":"4238e2f4950e48b62749d8f1723cc156b01165ed4c2af0108b851a079d78ee97"} Sep 17 02:04:05.241691 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:05.241631 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" podStartSLOduration=1.241614269 podStartE2EDuration="1.241614269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:04:05.240777417 +0000 UTC m=+1590.075829754" watchObservedRunningTime="2026-09-17 02:04:05.241614269 +0000 UTC m=+1590.076666603" Sep 17 02:04:05.845169 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:05.845138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:07.357590 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:07.357558 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.459651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.459610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.460428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.460401 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs"] Sep 17 02:04:11.462558 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.462533 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3f027f-29826840-6cqgs"] Sep 17 02:04:11.529427 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.529398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.586248 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.586180 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.679654 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.679628 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a21c85-8f4b-42d5-910f-2c802562962a" path="/var/lib/kubelet/pods/97a21c85-8f4b-42d5-910f-2c802562962a/volumes" Sep 17 02:04:11.686052 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.686027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.687249 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.687221 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4"] Sep 17 02:04:11.690005 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.689986 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3f027f-znbq4"] Sep 17 02:04:11.775643 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.775613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.908008 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.907979 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:11.967940 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:11.967910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:12.026097 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:12.026015 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:12.082310 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:12.082283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:12.137390 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:12.137362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:12.821970 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:12.821936 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:13.414640 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.414608 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:13.515593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.515560 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:13.611019 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.610984 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg"] Sep 17 02:04:13.613164 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.613134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:13.613620 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.613572 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e1c2c18-f65tg"] Sep 17 02:04:13.670826 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.670743 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:13.680248 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.680202 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b54f8810-216e-48fd-bfcf-c6bb7870bb60" path="/var/lib/kubelet/pods/b54f8810-216e-48fd-bfcf-c6bb7870bb60/volumes" Sep 17 02:04:13.680584 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:13.680571 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff85406-d413-476b-9309-19ec6483e2e5" path="/var/lib/kubelet/pods/bff85406-d413-476b-9309-19ec6483e2e5/volumes" Sep 17 02:04:14.602691 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:14.602656 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:14.887134 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:14.887052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:14.887134 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:14.887103 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:14.900115 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:14.900078 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:15.293692 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.293667 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:04:15.566319 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.566245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:15.567616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.567591 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" podUID="f4cdd27c-0bef-4906-9e95-22a98d3bfa00" containerName="main" containerID="cri-o://061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7" gracePeriod=45 Sep 17 02:04:15.579591 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.579562 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" podUID="ca9a3bea-202b-475e-afc6-42c7deb6b192" containerName="main" containerID="cri-o://d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf" gracePeriod=45 Sep 17 02:04:15.847176 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.847153 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 02:04:15.850307 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.850267 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 02:04:15.915183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-model-cache\") pod \"ca9a3bea-202b-475e-afc6-42c7deb6b192\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " Sep 17 02:04:15.915359 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915201 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-dshm\") pod \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " Sep 17 02:04:15.915359 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915288 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tmp-dir\") pod \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " Sep 17 02:04:15.915359 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915313 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tls-certs\") pod \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " Sep 17 02:04:15.915359 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915353 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-dshm\") pod \"ca9a3bea-202b-475e-afc6-42c7deb6b192\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " Sep 17 02:04:15.915623 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915446 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca9a3bea-202b-475e-afc6-42c7deb6b192" (UID: "ca9a3bea-202b-475e-afc6-42c7deb6b192"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.915623 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915466 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-model-cache\") pod \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " Sep 17 02:04:15.915623 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915499 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smfld\" (UniqueName: \"kubernetes.io/projected/ca9a3bea-202b-475e-afc6-42c7deb6b192-kube-api-access-smfld\") pod \"ca9a3bea-202b-475e-afc6-42c7deb6b192\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " Sep 17 02:04:15.915623 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915548 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj9l5\" (UniqueName: \"kubernetes.io/projected/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-kube-api-access-mj9l5\") pod \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " Sep 17 02:04:15.915623 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915576 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-tmp-dir\") pod \"ca9a3bea-202b-475e-afc6-42c7deb6b192\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915629 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-home\") pod \"ca9a3bea-202b-475e-afc6-42c7deb6b192\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915666 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-home\") pod \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\" (UID: \"f4cdd27c-0bef-4906-9e95-22a98d3bfa00\") " Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.915731 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9a3bea-202b-475e-afc6-42c7deb6b192-tls-certs\") pod \"ca9a3bea-202b-475e-afc6-42c7deb6b192\" (UID: \"ca9a3bea-202b-475e-afc6-42c7deb6b192\") " Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916024 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4cdd27c-0bef-4906-9e95-22a98d3bfa00" (UID: "f4cdd27c-0bef-4906-9e95-22a98d3bfa00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916020 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca9a3bea-202b-475e-afc6-42c7deb6b192" (UID: "ca9a3bea-202b-475e-afc6-42c7deb6b192"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916136 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916153 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916165 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916265 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4cdd27c-0bef-4906-9e95-22a98d3bfa00" (UID: "f4cdd27c-0bef-4906-9e95-22a98d3bfa00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916499 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-home" (OuterVolumeSpecName: "home") pod "f4cdd27c-0bef-4906-9e95-22a98d3bfa00" (UID: "f4cdd27c-0bef-4906-9e95-22a98d3bfa00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.916564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.916555 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-home" (OuterVolumeSpecName: "home") pod "ca9a3bea-202b-475e-afc6-42c7deb6b192" (UID: "ca9a3bea-202b-475e-afc6-42c7deb6b192"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.920905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.920882 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-kube-api-access-mj9l5" (OuterVolumeSpecName: "kube-api-access-mj9l5") pod "f4cdd27c-0bef-4906-9e95-22a98d3bfa00" (UID: "f4cdd27c-0bef-4906-9e95-22a98d3bfa00"). InnerVolumeSpecName "kube-api-access-mj9l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:04:15.921056 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.920885 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9a3bea-202b-475e-afc6-42c7deb6b192-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca9a3bea-202b-475e-afc6-42c7deb6b192" (UID: "ca9a3bea-202b-475e-afc6-42c7deb6b192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:15.921056 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.920890 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-dshm" (OuterVolumeSpecName: "dshm") pod "f4cdd27c-0bef-4906-9e95-22a98d3bfa00" (UID: "f4cdd27c-0bef-4906-9e95-22a98d3bfa00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:15.921056 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.920982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4cdd27c-0bef-4906-9e95-22a98d3bfa00" (UID: "f4cdd27c-0bef-4906-9e95-22a98d3bfa00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:04:15.921056 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.921032 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a3bea-202b-475e-afc6-42c7deb6b192-kube-api-access-smfld" (OuterVolumeSpecName: "kube-api-access-smfld") pod "ca9a3bea-202b-475e-afc6-42c7deb6b192" (UID: "ca9a3bea-202b-475e-afc6-42c7deb6b192"). InnerVolumeSpecName "kube-api-access-smfld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:04:15.922544 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:15.922521 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-dshm" (OuterVolumeSpecName: "dshm") pod "ca9a3bea-202b-475e-afc6-42c7deb6b192" (UID: "ca9a3bea-202b-475e-afc6-42c7deb6b192"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:04:16.017385 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017357 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj9l5\" (UniqueName: \"kubernetes.io/projected/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-kube-api-access-mj9l5\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017385 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017378 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017385 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017387 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017395 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9a3bea-202b-475e-afc6-42c7deb6b192-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017407 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017418 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017426 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cdd27c-0bef-4906-9e95-22a98d3bfa00-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017434 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9a3bea-202b-475e-afc6-42c7deb6b192-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.017606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.017441 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfld\" (UniqueName: \"kubernetes.io/projected/ca9a3bea-202b-475e-afc6-42c7deb6b192-kube-api-access-smfld\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:04:16.262473 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.262435 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:16.289309 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.289281 2582 generic.go:356] "Generic (PLEG): container finished" podID="ca9a3bea-202b-475e-afc6-42c7deb6b192" containerID="d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf" exitCode=0 Sep 17 02:04:16.289445 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.289323 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" Sep 17 02:04:16.289445 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.289373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" event={"ID":"ca9a3bea-202b-475e-afc6-42c7deb6b192","Type":"ContainerDied","Data":"d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf"} Sep 17 02:04:16.289445 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.289416 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66" event={"ID":"ca9a3bea-202b-475e-afc6-42c7deb6b192","Type":"ContainerDied","Data":"ff639236d047f90dc3a60579344be4bcd219263fdf1efb7671e27ef21e5a8f13"} Sep 17 02:04:16.289601 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.289444 2582 scope.go:117] "RemoveContainer" containerID="d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf" Sep 17 02:04:16.290882 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.290859 2582 generic.go:356] "Generic (PLEG): container finished" podID="f4cdd27c-0bef-4906-9e95-22a98d3bfa00" containerID="061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7" exitCode=0 Sep 17 02:04:16.290999 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.290901 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" Sep 17 02:04:16.290999 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.290944 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" event={"ID":"f4cdd27c-0bef-4906-9e95-22a98d3bfa00","Type":"ContainerDied","Data":"061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7"} Sep 17 02:04:16.290999 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.290979 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp" event={"ID":"f4cdd27c-0bef-4906-9e95-22a98d3bfa00","Type":"ContainerDied","Data":"aa732318b407efe226ee60eca36d9d6e72b1a1057e2be2f925687ec25370baa1"} Sep 17 02:04:16.299377 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.299363 2582 scope.go:117] "RemoveContainer" containerID="d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf" Sep 17 02:04:16.299636 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:04:16.299617 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf\": container with ID starting with d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf not found: ID does not exist" containerID="d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf" Sep 17 02:04:16.299709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.299642 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf"} err="failed to get container status \"d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf\": rpc error: code = NotFound desc = could not find container \"d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf\": container with ID starting with d0bbdbff9e348a8fc07def052d68917cd775aa14b1dd944edce8c28c7a2360bf not found: ID does not exist" Sep 17 02:04:16.299709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.299660 2582 scope.go:117] "RemoveContainer" containerID="061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7" Sep 17 02:04:16.308520 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.308504 2582 scope.go:117] "RemoveContainer" containerID="061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7" Sep 17 02:04:16.308774 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:04:16.308757 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7\": container with ID starting with 061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7 not found: ID does not exist" containerID="061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7" Sep 17 02:04:16.308824 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.308780 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7"} err="failed to get container status \"061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7\": rpc error: code = NotFound desc = could not find container \"061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7\": container with ID starting with 061ed861024dc73584c832b406303c3382bca531dec8f59508a712c61e1b47b7 not found: ID does not exist" Sep 17 02:04:16.315586 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.315566 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp"] Sep 17 02:04:16.319299 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.319280 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/premium-w3-3f027f-kserve-7c765c7955-tkjhp"] Sep 17 02:04:16.329387 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.329365 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66"] Sep 17 02:04:16.338507 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.338486 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3f027f/simulator-w3-3f027f-kserve-5fff47f766-bnr66"] Sep 17 02:04:16.910934 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:16.910861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:17.127278 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:17.127243 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:17.614517 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:17.614483 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:17.680489 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:17.680458 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9a3bea-202b-475e-afc6-42c7deb6b192" path="/var/lib/kubelet/pods/ca9a3bea-202b-475e-afc6-42c7deb6b192/volumes" Sep 17 02:04:17.681081 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:17.681051 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4cdd27c-0bef-4906-9e95-22a98d3bfa00" path="/var/lib/kubelet/pods/f4cdd27c-0bef-4906-9e95-22a98d3bfa00/volumes" Sep 17 02:04:18.388806 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:18.388767 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:18.720532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:18.720495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:18.817700 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:18.817653 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:18.990396 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:18.990312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:21.940320 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:21.940288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:34.341819 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:34.341773 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:35.803043 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:35.803012 2582 scope.go:117] "RemoveContainer" containerID="ffa445f13f379dcf34ea31a6d51d35bcd42d0ffda1f74eef881da3bc0d77ecc8" Sep 17 02:04:37.469690 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:37.469650 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:37.570166 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:37.570134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:37.702467 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:37.702439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:37.773183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:37.773100 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:04:43.218565 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:43.218531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:58.209291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:58.209257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:58.853180 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:58.853146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:04:59.871144 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:04:59.871110 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:05:01.036240 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:01.036175 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:01.135493 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:01.135463 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:04.471180 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:04.471137 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:04.536500 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:04.536459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:04.757400 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:04.757317 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:06.090645 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:06.090614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:06.189032 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:06.188996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:06.289422 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:06.289382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:06.997485 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:06.997450 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:07.095317 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:07.095279 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:07.195564 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:07.195529 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:08.031726 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.031689 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:08.151649 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.151622 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr"] Sep 17 02:05:08.152052 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.152039 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9a3bea-202b-475e-afc6-42c7deb6b192" containerName="main" Sep 17 02:05:08.152099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.152054 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9a3bea-202b-475e-afc6-42c7deb6b192" containerName="main" Sep 17 02:05:08.152099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.152075 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4cdd27c-0bef-4906-9e95-22a98d3bfa00" containerName="main" Sep 17 02:05:08.152099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.152081 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4cdd27c-0bef-4906-9e95-22a98d3bfa00" containerName="main" Sep 17 02:05:08.152197 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.152150 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4cdd27c-0bef-4906-9e95-22a98d3bfa00" containerName="main" Sep 17 02:05:08.152197 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.152158 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9a3bea-202b-475e-afc6-42c7deb6b192" containerName="main" Sep 17 02:05:08.154358 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.154342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:08.156795 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.156776 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:05:08.156795 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.156789 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:05:08.162707 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.162688 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr"] Sep 17 02:05:08.165466 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.165442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c8dbd3-f3df-4fd1-a204-bdddf1b14172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr\" (UID: \"25c8dbd3-f3df-4fd1-a204-bdddf1b14172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:08.266767 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.266744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c8dbd3-f3df-4fd1-a204-bdddf1b14172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr\" (UID: \"25c8dbd3-f3df-4fd1-a204-bdddf1b14172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:08.267327 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.267311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c8dbd3-f3df-4fd1-a204-bdddf1b14172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr\" (UID: \"25c8dbd3-f3df-4fd1-a204-bdddf1b14172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:08.466629 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.466608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:08.514005 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.513945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:08.613885 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.613848 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:08.737318 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:08.737295 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr"] Sep 17 02:05:08.738831 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:05:08.738800 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c8dbd3_f3df_4fd1_a204_bdddf1b14172.slice/crio-27ba6d0292e0bfe8588eeb51238054757d5bb520070826d7d172d8b6f0d06c8b WatchSource:0}: Error finding container 27ba6d0292e0bfe8588eeb51238054757d5bb520070826d7d172d8b6f0d06c8b: Status 404 returned error can't find the container with id 27ba6d0292e0bfe8588eeb51238054757d5bb520070826d7d172d8b6f0d06c8b Sep 17 02:05:09.338648 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:09.338574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:09.593262 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:09.593184 2582 generic.go:356] "Generic (PLEG): container finished" podID="25c8dbd3-f3df-4fd1-a204-bdddf1b14172" containerID="55b1281dbd2096ea1671779898019fd9713a42aa35ca3a6fea56fc74508099ee" exitCode=0 Sep 17 02:05:09.593262 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:09.593247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" event={"ID":"25c8dbd3-f3df-4fd1-a204-bdddf1b14172","Type":"ContainerDied","Data":"55b1281dbd2096ea1671779898019fd9713a42aa35ca3a6fea56fc74508099ee"} Sep 17 02:05:09.593398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:09.593299 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" event={"ID":"25c8dbd3-f3df-4fd1-a204-bdddf1b14172","Type":"ContainerStarted","Data":"27ba6d0292e0bfe8588eeb51238054757d5bb520070826d7d172d8b6f0d06c8b"} Sep 17 02:05:10.040736 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:10.040707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:10.738579 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:10.738554 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:10.773094 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:10.773061 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:10.790229 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:10.790187 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c8dbd3-f3df-4fd1-a204-bdddf1b14172-maas-api-service-ca\") pod \"25c8dbd3-f3df-4fd1-a204-bdddf1b14172\" (UID: \"25c8dbd3-f3df-4fd1-a204-bdddf1b14172\") " Sep 17 02:05:10.790552 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:10.790527 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c8dbd3-f3df-4fd1-a204-bdddf1b14172-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25c8dbd3-f3df-4fd1-a204-bdddf1b14172" (UID: "25c8dbd3-f3df-4fd1-a204-bdddf1b14172"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:05:10.891479 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:10.891418 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c8dbd3-f3df-4fd1-a204-bdddf1b14172-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:11.442280 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:11.442246 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:11.606309 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:11.606277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" event={"ID":"25c8dbd3-f3df-4fd1-a204-bdddf1b14172","Type":"ContainerDied","Data":"27ba6d0292e0bfe8588eeb51238054757d5bb520070826d7d172d8b6f0d06c8b"} Sep 17 02:05:11.606465 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:11.606324 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr" Sep 17 02:05:11.606465 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:11.606355 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ba6d0292e0bfe8588eeb51238054757d5bb520070826d7d172d8b6f0d06c8b" Sep 17 02:05:12.021855 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:12.021821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:12.837980 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:12.837945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:13.012496 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:13.012464 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc"] Sep 17 02:05:13.029497 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:13.029467 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd"] Sep 17 02:05:13.041228 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:13.041179 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv"] Sep 17 02:05:13.048758 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:13.048732 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7"] Sep 17 02:05:13.106852 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:13.106776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:14.462616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:14.462579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:14.529013 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:14.528985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:15.195525 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:15.195489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:20.744335 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:20.744297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:20.804597 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:20.804567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:20.875453 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:20.875423 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:20.976020 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:20.975992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:23.822756 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:23.822718 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:23.843077 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:23.843051 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp"] Sep 17 02:05:23.843291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:23.843267 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" podUID="0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" containerName="payload-processing" containerID="cri-o://413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179" gracePeriod=30 Sep 17 02:05:23.891759 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:23.891729 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:23.993039 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:23.993007 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:24.027973 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.027943 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr"] Sep 17 02:05:24.030569 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.030544 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e564e1-4ztfr"] Sep 17 02:05:24.076830 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.076788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:24.098847 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.098827 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 02:05:24.205644 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.205616 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljn2g\" (UniqueName: \"kubernetes.io/projected/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-kube-api-access-ljn2g\") pod \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " Sep 17 02:05:24.206022 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.205996 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-plugins-config-volume\") pod \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\" (UID: \"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2\") " Sep 17 02:05:24.206567 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.206482 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" (UID: "0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:05:24.206720 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.206694 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:24.207944 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.207924 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-kube-api-access-ljn2g" (OuterVolumeSpecName: "kube-api-access-ljn2g") pod "0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" (UID: "0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2"). InnerVolumeSpecName "kube-api-access-ljn2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:05:24.307417 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.307398 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn2g\" (UniqueName: \"kubernetes.io/projected/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2-kube-api-access-ljn2g\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:24.392891 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.392821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:24.528543 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.528518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:24.682308 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.682247 2582 generic.go:356] "Generic (PLEG): container finished" podID="0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" containerID="413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179" exitCode=0 Sep 17 02:05:24.682308 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.682286 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" Sep 17 02:05:24.682437 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.682323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" event={"ID":"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2","Type":"ContainerDied","Data":"413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179"} Sep 17 02:05:24.682437 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.682366 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp" event={"ID":"0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2","Type":"ContainerDied","Data":"a6fc910b94b1196008bd8a5ae18ce08d6bf8dc8c3f5ca09a5589f855705a7f8b"} Sep 17 02:05:24.682437 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.682396 2582 scope.go:117] "RemoveContainer" containerID="413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179" Sep 17 02:05:24.692266 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.692251 2582 scope.go:117] "RemoveContainer" containerID="413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179" Sep 17 02:05:24.692546 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:05:24.692527 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179\": container with ID starting with 413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179 not found: ID does not exist" containerID="413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179" Sep 17 02:05:24.692601 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.692554 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179"} err="failed to get container status \"413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179\": rpc error: code = NotFound desc = could not find container \"413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179\": container with ID starting with 413788cb7bf71a5f0fac615d92bdce8f14b029dc310f35a0094422c4da8a8179 not found: ID does not exist" Sep 17 02:05:24.703642 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.703621 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp"] Sep 17 02:05:24.707923 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:24.707906 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e564e1-7d588459b7-lrxlp"] Sep 17 02:05:25.330542 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:25.330504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:25.680571 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:25.680499 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" path="/var/lib/kubelet/pods/0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2/volumes" Sep 17 02:05:25.680858 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:25.680844 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c8dbd3-f3df-4fd1-a204-bdddf1b14172" path="/var/lib/kubelet/pods/25c8dbd3-f3df-4fd1-a204-bdddf1b14172/volumes" Sep 17 02:05:26.042122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:26.042096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:26.281774 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:26.281743 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:05:26.782516 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:26.782485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:27.337201 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:27.337153 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:28.032509 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.032468 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" podUID="52fd9f12-461d-4f9f-ad9c-8cc073edd933" containerName="main" containerID="cri-o://2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601" gracePeriod=45 Sep 17 02:05:28.053625 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.053595 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" podUID="e8069c44-7652-42a2-a27f-bf7670206814" containerName="main" containerID="cri-o://80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93" gracePeriod=45 Sep 17 02:05:28.067054 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.067029 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" podUID="63d24bc1-2dab-44fd-b5d7-d529d29a443b" containerName="main" containerID="cri-o://aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769" gracePeriod=45 Sep 17 02:05:28.124526 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:05:28.124499 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63d24bc1_2dab_44fd_b5d7_d529d29a443b.slice/crio-conmon-aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769.scope\": RecentStats: unable to find data in memory cache]" Sep 17 02:05:28.318730 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.318703 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:05:28.361949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.361926 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:05:28.365731 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.365707 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 02:05:28.446893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.446863 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-dshm\") pod \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " Sep 17 02:05:28.446893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.446892 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-home\") pod \"e8069c44-7652-42a2-a27f-bf7670206814\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.446911 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-model-cache\") pod \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.446931 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-home\") pod \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.446961 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ssrr\" (UniqueName: \"kubernetes.io/projected/63d24bc1-2dab-44fd-b5d7-d529d29a443b-kube-api-access-4ssrr\") pod \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.446977 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tls-certs\") pod \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447004 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tmp-dir\") pod \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447028 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ptrq\" (UniqueName: \"kubernetes.io/projected/52fd9f12-461d-4f9f-ad9c-8cc073edd933-kube-api-access-8ptrq\") pod \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " Sep 17 02:05:28.447122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447080 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-home\") pod \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447143 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8069c44-7652-42a2-a27f-bf7670206814-tls-certs\") pod \"e8069c44-7652-42a2-a27f-bf7670206814\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447147 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-model-cache" (OuterVolumeSpecName: "model-cache") pod "52fd9f12-461d-4f9f-ad9c-8cc073edd933" (UID: "52fd9f12-461d-4f9f-ad9c-8cc073edd933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447158 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-home" (OuterVolumeSpecName: "home") pod "e8069c44-7652-42a2-a27f-bf7670206814" (UID: "e8069c44-7652-42a2-a27f-bf7670206814"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447185 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tls-certs\") pod \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447249 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-dshm\") pod \"e8069c44-7652-42a2-a27f-bf7670206814\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447279 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-model-cache\") pod \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447306 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-model-cache\") pod \"e8069c44-7652-42a2-a27f-bf7670206814\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447332 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxgwf\" (UniqueName: \"kubernetes.io/projected/e8069c44-7652-42a2-a27f-bf7670206814-kube-api-access-rxgwf\") pod \"e8069c44-7652-42a2-a27f-bf7670206814\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447323 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-home" (OuterVolumeSpecName: "home") pod "52fd9f12-461d-4f9f-ad9c-8cc073edd933" (UID: "52fd9f12-461d-4f9f-ad9c-8cc073edd933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447367 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-dshm\") pod \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\" (UID: \"63d24bc1-2dab-44fd-b5d7-d529d29a443b\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447379 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63d24bc1-2dab-44fd-b5d7-d529d29a443b" (UID: "63d24bc1-2dab-44fd-b5d7-d529d29a443b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447422 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-tmp-dir\") pod \"e8069c44-7652-42a2-a27f-bf7670206814\" (UID: \"e8069c44-7652-42a2-a27f-bf7670206814\") " Sep 17 02:05:28.447523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447473 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tmp-dir\") pod \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\" (UID: \"52fd9f12-461d-4f9f-ad9c-8cc073edd933\") " Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447532 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-model-cache" (OuterVolumeSpecName: "model-cache") pod "63d24bc1-2dab-44fd-b5d7-d529d29a443b" (UID: "63d24bc1-2dab-44fd-b5d7-d529d29a443b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447870 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447889 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447904 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447916 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.447928 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.448179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.448135 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52fd9f12-461d-4f9f-ad9c-8cc073edd933" (UID: "52fd9f12-461d-4f9f-ad9c-8cc073edd933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.448554 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.448333 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8069c44-7652-42a2-a27f-bf7670206814" (UID: "e8069c44-7652-42a2-a27f-bf7670206814"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.448775 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.448695 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8069c44-7652-42a2-a27f-bf7670206814" (UID: "e8069c44-7652-42a2-a27f-bf7670206814"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.448775 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.448755 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-home" (OuterVolumeSpecName: "home") pod "63d24bc1-2dab-44fd-b5d7-d529d29a443b" (UID: "63d24bc1-2dab-44fd-b5d7-d529d29a443b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.449733 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.449701 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fd9f12-461d-4f9f-ad9c-8cc073edd933-kube-api-access-8ptrq" (OuterVolumeSpecName: "kube-api-access-8ptrq") pod "52fd9f12-461d-4f9f-ad9c-8cc073edd933" (UID: "52fd9f12-461d-4f9f-ad9c-8cc073edd933"). InnerVolumeSpecName "kube-api-access-8ptrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:05:28.450143 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450068 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-dshm" (OuterVolumeSpecName: "dshm") pod "e8069c44-7652-42a2-a27f-bf7670206814" (UID: "e8069c44-7652-42a2-a27f-bf7670206814"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.450143 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450129 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63d24bc1-2dab-44fd-b5d7-d529d29a443b" (UID: "63d24bc1-2dab-44fd-b5d7-d529d29a443b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:05:28.450385 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450291 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-dshm" (OuterVolumeSpecName: "dshm") pod "52fd9f12-461d-4f9f-ad9c-8cc073edd933" (UID: "52fd9f12-461d-4f9f-ad9c-8cc073edd933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.450385 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450320 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d24bc1-2dab-44fd-b5d7-d529d29a443b-kube-api-access-4ssrr" (OuterVolumeSpecName: "kube-api-access-4ssrr") pod "63d24bc1-2dab-44fd-b5d7-d529d29a443b" (UID: "63d24bc1-2dab-44fd-b5d7-d529d29a443b"). InnerVolumeSpecName "kube-api-access-4ssrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:05:28.450473 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450437 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8069c44-7652-42a2-a27f-bf7670206814-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8069c44-7652-42a2-a27f-bf7670206814" (UID: "e8069c44-7652-42a2-a27f-bf7670206814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:05:28.450693 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450675 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8069c44-7652-42a2-a27f-bf7670206814-kube-api-access-rxgwf" (OuterVolumeSpecName: "kube-api-access-rxgwf") pod "e8069c44-7652-42a2-a27f-bf7670206814" (UID: "e8069c44-7652-42a2-a27f-bf7670206814"). InnerVolumeSpecName "kube-api-access-rxgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:05:28.450752 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.450739 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52fd9f12-461d-4f9f-ad9c-8cc073edd933" (UID: "52fd9f12-461d-4f9f-ad9c-8cc073edd933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:05:28.451311 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.451295 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-dshm" (OuterVolumeSpecName: "dshm") pod "63d24bc1-2dab-44fd-b5d7-d529d29a443b" (UID: "63d24bc1-2dab-44fd-b5d7-d529d29a443b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.549428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549371 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549395 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ssrr\" (UniqueName: \"kubernetes.io/projected/63d24bc1-2dab-44fd-b5d7-d529d29a443b-kube-api-access-4ssrr\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549406 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549416 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptrq\" (UniqueName: \"kubernetes.io/projected/52fd9f12-461d-4f9f-ad9c-8cc073edd933-kube-api-access-8ptrq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549424 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549431 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8069c44-7652-42a2-a27f-bf7670206814-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549439 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d24bc1-2dab-44fd-b5d7-d529d29a443b-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549446 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549454 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549461 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgwf\" (UniqueName: \"kubernetes.io/projected/e8069c44-7652-42a2-a27f-bf7670206814-kube-api-access-rxgwf\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549468 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d24bc1-2dab-44fd-b5d7-d529d29a443b-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549476 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8069c44-7652-42a2-a27f-bf7670206814-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.549684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.549482 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fd9f12-461d-4f9f-ad9c-8cc073edd933-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.638405 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.638376 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" podUID="ae1edbb2-d236-409a-82f9-6038b13b7970" containerName="main" containerID="cri-o://d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0" gracePeriod=45 Sep 17 02:05:28.706191 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.706168 2582 generic.go:356] "Generic (PLEG): container finished" podID="63d24bc1-2dab-44fd-b5d7-d529d29a443b" containerID="aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769" exitCode=0 Sep 17 02:05:28.706295 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.706202 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" Sep 17 02:05:28.706295 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.706249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" event={"ID":"63d24bc1-2dab-44fd-b5d7-d529d29a443b","Type":"ContainerDied","Data":"aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769"} Sep 17 02:05:28.706295 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.706287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7" event={"ID":"63d24bc1-2dab-44fd-b5d7-d529d29a443b","Type":"ContainerDied","Data":"18cb34f771a717d6c68f5ee971aced6f99a45844b20d727258e0a287e6e4d76b"} Sep 17 02:05:28.706435 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.706310 2582 scope.go:117] "RemoveContainer" containerID="aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769" Sep 17 02:05:28.707699 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.707678 2582 generic.go:356] "Generic (PLEG): container finished" podID="52fd9f12-461d-4f9f-ad9c-8cc073edd933" containerID="2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601" exitCode=0 Sep 17 02:05:28.707773 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.707734 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" Sep 17 02:05:28.707831 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.707764 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" event={"ID":"52fd9f12-461d-4f9f-ad9c-8cc073edd933","Type":"ContainerDied","Data":"2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601"} Sep 17 02:05:28.707831 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.707803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc" event={"ID":"52fd9f12-461d-4f9f-ad9c-8cc073edd933","Type":"ContainerDied","Data":"951e5bd69d61ac2426220b215b310b1f31828e17399a422fa7f9e69a2770a998"} Sep 17 02:05:28.709155 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.709134 2582 generic.go:356] "Generic (PLEG): container finished" podID="e8069c44-7652-42a2-a27f-bf7670206814" containerID="80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93" exitCode=0 Sep 17 02:05:28.709389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.709178 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" Sep 17 02:05:28.709389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.709191 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" event={"ID":"e8069c44-7652-42a2-a27f-bf7670206814","Type":"ContainerDied","Data":"80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93"} Sep 17 02:05:28.709389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.709236 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd" event={"ID":"e8069c44-7652-42a2-a27f-bf7670206814","Type":"ContainerDied","Data":"4238e2f4950e48b62749d8f1723cc156b01165ed4c2af0108b851a079d78ee97"} Sep 17 02:05:28.733297 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.733276 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7"] Sep 17 02:05:28.736757 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.736737 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/simulator-w0-e564e1-kserve-94d859b59-tckb7"] Sep 17 02:05:28.749745 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.749642 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc"] Sep 17 02:05:28.750516 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.750494 2582 scope.go:117] "RemoveContainer" containerID="aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769" Sep 17 02:05:28.750824 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:05:28.750802 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769\": container with ID starting with aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769 not found: ID does not exist" containerID="aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769" Sep 17 02:05:28.750909 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.750833 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769"} err="failed to get container status \"aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769\": rpc error: code = NotFound desc = could not find container \"aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769\": container with ID starting with aa2999c21789524448e31f1c360827f24e185afd1096fab43e0677a15a582769 not found: ID does not exist" Sep 17 02:05:28.750909 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.750850 2582 scope.go:117] "RemoveContainer" containerID="2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601" Sep 17 02:05:28.752558 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.752534 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/distinct-2-w0-e564e1-kserve-6544c47d9d-nk8dc"] Sep 17 02:05:28.762578 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.762555 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd"] Sep 17 02:05:28.762772 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.762758 2582 scope.go:117] "RemoveContainer" containerID="2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601" Sep 17 02:05:28.763794 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:05:28.763771 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601\": container with ID starting with 2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601 not found: ID does not exist" containerID="2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601" Sep 17 02:05:28.763887 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.763802 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601"} err="failed to get container status \"2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601\": rpc error: code = NotFound desc = could not find container \"2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601\": container with ID starting with 2bdbe0b87eb7fee3219f8ec363b5a749308c0744bef7114c603d1643b4b3e601 not found: ID does not exist" Sep 17 02:05:28.763887 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.763818 2582 scope.go:117] "RemoveContainer" containerID="80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93" Sep 17 02:05:28.765846 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.765828 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/embedding-w0-e564e1-kserve-6444745b58-vflwd"] Sep 17 02:05:28.772494 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.772468 2582 scope.go:117] "RemoveContainer" containerID="80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93" Sep 17 02:05:28.772800 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:05:28.772782 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93\": container with ID starting with 80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93 not found: ID does not exist" containerID="80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93" Sep 17 02:05:28.772879 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.772805 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93"} err="failed to get container status \"80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93\": rpc error: code = NotFound desc = could not find container \"80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93\": container with ID starting with 80a208101b1ed701a22d6d78e555b066a1b13535023965109ec862b6e0146c93 not found: ID does not exist" Sep 17 02:05:28.882257 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.882233 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 02:05:28.952463 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952434 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-dshm\") pod \"ae1edbb2-d236-409a-82f9-6038b13b7970\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " Sep 17 02:05:28.952621 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952591 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt4vb\" (UniqueName: \"kubernetes.io/projected/ae1edbb2-d236-409a-82f9-6038b13b7970-kube-api-access-nt4vb\") pod \"ae1edbb2-d236-409a-82f9-6038b13b7970\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " Sep 17 02:05:28.952684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952630 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1edbb2-d236-409a-82f9-6038b13b7970-tls-certs\") pod \"ae1edbb2-d236-409a-82f9-6038b13b7970\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " Sep 17 02:05:28.952684 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952653 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-model-cache\") pod \"ae1edbb2-d236-409a-82f9-6038b13b7970\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " Sep 17 02:05:28.952801 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-tmp-dir\") pod \"ae1edbb2-d236-409a-82f9-6038b13b7970\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " Sep 17 02:05:28.952801 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-home\") pod \"ae1edbb2-d236-409a-82f9-6038b13b7970\" (UID: \"ae1edbb2-d236-409a-82f9-6038b13b7970\") " Sep 17 02:05:28.952984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952911 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae1edbb2-d236-409a-82f9-6038b13b7970" (UID: "ae1edbb2-d236-409a-82f9-6038b13b7970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.952984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.952966 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae1edbb2-d236-409a-82f9-6038b13b7970" (UID: "ae1edbb2-d236-409a-82f9-6038b13b7970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.953122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.953007 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-home" (OuterVolumeSpecName: "home") pod "ae1edbb2-d236-409a-82f9-6038b13b7970" (UID: "ae1edbb2-d236-409a-82f9-6038b13b7970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:28.953122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.953087 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.953122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.953107 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:28.954670 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.954650 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1edbb2-d236-409a-82f9-6038b13b7970-kube-api-access-nt4vb" (OuterVolumeSpecName: "kube-api-access-nt4vb") pod "ae1edbb2-d236-409a-82f9-6038b13b7970" (UID: "ae1edbb2-d236-409a-82f9-6038b13b7970"). InnerVolumeSpecName "kube-api-access-nt4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:05:28.954670 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.954658 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1edbb2-d236-409a-82f9-6038b13b7970-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae1edbb2-d236-409a-82f9-6038b13b7970" (UID: "ae1edbb2-d236-409a-82f9-6038b13b7970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:05:28.954978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:28.954964 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-dshm" (OuterVolumeSpecName: "dshm") pod "ae1edbb2-d236-409a-82f9-6038b13b7970" (UID: "ae1edbb2-d236-409a-82f9-6038b13b7970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:05:29.053912 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.053886 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4vb\" (UniqueName: \"kubernetes.io/projected/ae1edbb2-d236-409a-82f9-6038b13b7970-kube-api-access-nt4vb\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:29.053912 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.053910 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae1edbb2-d236-409a-82f9-6038b13b7970-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:29.054262 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.053920 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:29.054262 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.053928 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae1edbb2-d236-409a-82f9-6038b13b7970-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:05:29.679758 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.679727 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52fd9f12-461d-4f9f-ad9c-8cc073edd933" path="/var/lib/kubelet/pods/52fd9f12-461d-4f9f-ad9c-8cc073edd933/volumes" Sep 17 02:05:29.680115 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.680101 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d24bc1-2dab-44fd-b5d7-d529d29a443b" path="/var/lib/kubelet/pods/63d24bc1-2dab-44fd-b5d7-d529d29a443b/volumes" Sep 17 02:05:29.680479 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.680466 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8069c44-7652-42a2-a27f-bf7670206814" path="/var/lib/kubelet/pods/e8069c44-7652-42a2-a27f-bf7670206814/volumes" Sep 17 02:05:29.718348 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.718318 2582 generic.go:356] "Generic (PLEG): container finished" podID="ae1edbb2-d236-409a-82f9-6038b13b7970" containerID="d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0" exitCode=0 Sep 17 02:05:29.718468 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.718357 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" Sep 17 02:05:29.718468 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.718386 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" event={"ID":"ae1edbb2-d236-409a-82f9-6038b13b7970","Type":"ContainerDied","Data":"d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0"} Sep 17 02:05:29.718468 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.718419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv" event={"ID":"ae1edbb2-d236-409a-82f9-6038b13b7970","Type":"ContainerDied","Data":"e437d1d1a34d5be1a21c0a32f0829c5b8da28178dbdbb8300cb759b3c704222c"} Sep 17 02:05:29.718468 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.718441 2582 scope.go:117] "RemoveContainer" containerID="d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0" Sep 17 02:05:29.727505 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.727489 2582 scope.go:117] "RemoveContainer" containerID="d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0" Sep 17 02:05:29.727771 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:05:29.727755 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0\": container with ID starting with d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0 not found: ID does not exist" containerID="d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0" Sep 17 02:05:29.727843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.727780 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0"} err="failed to get container status \"d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0\": rpc error: code = NotFound desc = could not find container \"d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0\": container with ID starting with d7f0a6d2f42afa9587ee9505654c39da1695b5ab2e000ca44fa4b1aa4a9ed1d0 not found: ID does not exist" Sep 17 02:05:29.734275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.734254 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv"] Sep 17 02:05:29.738297 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:29.738277 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e564e1/premium-w0-e564e1-kserve-745c9764f4-fpvfv"] Sep 17 02:05:30.131856 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:30.131828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:30.413070 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:30.412985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:30.512169 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:30.512138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:30.687811 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:30.687722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:31.469812 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:31.469773 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:31.681228 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:31.681184 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1edbb2-d236-409a-82f9-6038b13b7970" path="/var/lib/kubelet/pods/ae1edbb2-d236-409a-82f9-6038b13b7970/volumes" Sep 17 02:05:31.767367 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:31.767280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:48.566921 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:48.566888 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:05:49.634411 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:49.634372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:49.735840 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:49.735804 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:50.767402 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:50.767366 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:52.676465 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:52.676432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:56.309053 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:56.309016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:58.385324 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:58.385248 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:58.485226 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:58.485185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:58.736801 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:58.736765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:59.047516 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:59.047429 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:05:59.182429 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:05:59.182399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:01.877191 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:01.877157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:02.157994 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:02.157903 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:02.255866 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:02.255830 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:02.759887 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:02.759803 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:03.556903 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:03.556866 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:04.373669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:04.373624 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:04.474509 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:04.474467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:04.903998 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:04.903963 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:06:05.238798 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:05.238765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:05.263748 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:05.263714 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d"] Sep 17 02:06:05.338962 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:05.338923 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:05.397290 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:05.397253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:05.851965 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:05.851922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:05.960708 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:05.960674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:07.167781 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:07.167725 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:15.969878 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:15.969839 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:06:19.168513 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168477 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg"] Sep 17 02:06:19.168905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168889 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c8dbd3-f3df-4fd1-a204-bdddf1b14172" containerName="revoke-keys" Sep 17 02:06:19.168905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168901 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c8dbd3-f3df-4fd1-a204-bdddf1b14172" containerName="revoke-keys" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168913 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8069c44-7652-42a2-a27f-bf7670206814" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168919 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8069c44-7652-42a2-a27f-bf7670206814" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168928 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d24bc1-2dab-44fd-b5d7-d529d29a443b" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168933 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d24bc1-2dab-44fd-b5d7-d529d29a443b" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168944 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" containerName="payload-processing" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168949 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" containerName="payload-processing" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168957 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52fd9f12-461d-4f9f-ad9c-8cc073edd933" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168962 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="52fd9f12-461d-4f9f-ad9c-8cc073edd933" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168972 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1edbb2-d236-409a-82f9-6038b13b7970" containerName="main" Sep 17 02:06:19.168978 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.168977 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1edbb2-d236-409a-82f9-6038b13b7970" containerName="main" Sep 17 02:06:19.169312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.169034 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c8dbd3-f3df-4fd1-a204-bdddf1b14172" containerName="revoke-keys" Sep 17 02:06:19.169312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.169045 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3d2bb3-ceed-4c65-8203-e4008b9ac0c2" containerName="payload-processing" Sep 17 02:06:19.169312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.169050 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8069c44-7652-42a2-a27f-bf7670206814" containerName="main" Sep 17 02:06:19.169312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.169057 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d24bc1-2dab-44fd-b5d7-d529d29a443b" containerName="main" Sep 17 02:06:19.169312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.169064 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="52fd9f12-461d-4f9f-ad9c-8cc073edd933" containerName="main" Sep 17 02:06:19.169312 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.169069 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1edbb2-d236-409a-82f9-6038b13b7970" containerName="main" Sep 17 02:06:19.172268 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.172252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.175586 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.175560 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"default-dockercfg-47wrb\"" Sep 17 02:06:19.175716 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.175646 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"kube-root-ca.crt\"" Sep 17 02:06:19.175716 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.175573 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"openshift-service-ca.crt\"" Sep 17 02:06:19.175846 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.175727 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-67a23a\"/\"sub-test-model-67a23a-kserve-self-signed-certs\"" Sep 17 02:06:19.181097 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.181067 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg"] Sep 17 02:06:19.199897 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.199870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:19.325986 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.325962 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxsq\" (UniqueName: \"kubernetes.io/projected/760ec0bd-4f8f-499b-8857-d303e689745a-kube-api-access-9dxsq\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.326099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.325997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-tmp-dir\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.326099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.326020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-home\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.326099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.326094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-dshm\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.326250 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.326159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760ec0bd-4f8f-499b-8857-d303e689745a-tls-certs\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.326250 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.326194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-model-cache\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-home\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-dshm\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427457 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760ec0bd-4f8f-499b-8857-d303e689745a-tls-certs\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427457 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-model-cache\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427457 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxsq\" (UniqueName: \"kubernetes.io/projected/760ec0bd-4f8f-499b-8857-d303e689745a-kube-api-access-9dxsq\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427457 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-tmp-dir\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427673 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427648 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-home\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-tmp-dir\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.427775 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.427747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-model-cache\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.429448 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.429430 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-dshm\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.429708 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.429692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760ec0bd-4f8f-499b-8857-d303e689745a-tls-certs\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.434739 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.434709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxsq\" (UniqueName: \"kubernetes.io/projected/760ec0bd-4f8f-499b-8857-d303e689745a-kube-api-access-9dxsq\") pod \"sub-test-model-67a23a-kserve-7f76f8d86c-784mg\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.482785 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.482763 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:19.754905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.754879 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg"] Sep 17 02:06:19.756919 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:06:19.756880 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod760ec0bd_4f8f_499b_8857_d303e689745a.slice/crio-012854fa412a77d024f9f1e3d7bd5943d72f8f0698d456c0b962313402637d01 WatchSource:0}: Error finding container 012854fa412a77d024f9f1e3d7bd5943d72f8f0698d456c0b962313402637d01: Status 404 returned error can't find the container with id 012854fa412a77d024f9f1e3d7bd5943d72f8f0698d456c0b962313402637d01 Sep 17 02:06:19.997897 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.997863 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" event={"ID":"760ec0bd-4f8f-499b-8857-d303e689745a","Type":"ContainerStarted","Data":"7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e"} Sep 17 02:06:19.997897 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:19.997910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" event={"ID":"760ec0bd-4f8f-499b-8857-d303e689745a","Type":"ContainerStarted","Data":"012854fa412a77d024f9f1e3d7bd5943d72f8f0698d456c0b962313402637d01"} Sep 17 02:06:20.015571 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.015480 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" podStartSLOduration=1.015460304 podStartE2EDuration="1.015460304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:06:20.012129692 +0000 UTC m=+1724.847182006" watchObservedRunningTime="2026-09-17 02:06:20.015460304 +0000 UTC m=+1724.850512639" Sep 17 02:06:20.283646 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.283558 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" podUID="700cdeee-8acf-4668-96ff-61c7c16fb10d" containerName="main" containerID="cri-o://089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86" gracePeriod=45 Sep 17 02:06:20.530561 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.530536 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:06:20.637780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.637700 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-home\") pod \"700cdeee-8acf-4668-96ff-61c7c16fb10d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " Sep 17 02:06:20.637780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.637739 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-model-cache\") pod \"700cdeee-8acf-4668-96ff-61c7c16fb10d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " Sep 17 02:06:20.637780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.637770 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700cdeee-8acf-4668-96ff-61c7c16fb10d-tls-certs\") pod \"700cdeee-8acf-4668-96ff-61c7c16fb10d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " Sep 17 02:06:20.638049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.637926 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtfrq\" (UniqueName: \"kubernetes.io/projected/700cdeee-8acf-4668-96ff-61c7c16fb10d-kube-api-access-dtfrq\") pod \"700cdeee-8acf-4668-96ff-61c7c16fb10d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " Sep 17 02:06:20.638049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.637971 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-tmp-dir\") pod \"700cdeee-8acf-4668-96ff-61c7c16fb10d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " Sep 17 02:06:20.638049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.637995 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-dshm\") pod \"700cdeee-8acf-4668-96ff-61c7c16fb10d\" (UID: \"700cdeee-8acf-4668-96ff-61c7c16fb10d\") " Sep 17 02:06:20.638049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.638001 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-model-cache" (OuterVolumeSpecName: "model-cache") pod "700cdeee-8acf-4668-96ff-61c7c16fb10d" (UID: "700cdeee-8acf-4668-96ff-61c7c16fb10d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:20.638049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.638011 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-home" (OuterVolumeSpecName: "home") pod "700cdeee-8acf-4668-96ff-61c7c16fb10d" (UID: "700cdeee-8acf-4668-96ff-61c7c16fb10d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:20.638303 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.638229 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "700cdeee-8acf-4668-96ff-61c7c16fb10d" (UID: "700cdeee-8acf-4668-96ff-61c7c16fb10d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:20.638355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.638346 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:06:20.638418 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.638363 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:06:20.638418 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.638377 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:06:20.640160 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.640123 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700cdeee-8acf-4668-96ff-61c7c16fb10d-kube-api-access-dtfrq" (OuterVolumeSpecName: "kube-api-access-dtfrq") pod "700cdeee-8acf-4668-96ff-61c7c16fb10d" (UID: "700cdeee-8acf-4668-96ff-61c7c16fb10d"). InnerVolumeSpecName "kube-api-access-dtfrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:06:20.640160 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.640135 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700cdeee-8acf-4668-96ff-61c7c16fb10d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "700cdeee-8acf-4668-96ff-61c7c16fb10d" (UID: "700cdeee-8acf-4668-96ff-61c7c16fb10d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:06:20.640160 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.640134 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-dshm" (OuterVolumeSpecName: "dshm") pod "700cdeee-8acf-4668-96ff-61c7c16fb10d" (UID: "700cdeee-8acf-4668-96ff-61c7c16fb10d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:06:20.739612 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.739583 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfrq\" (UniqueName: \"kubernetes.io/projected/700cdeee-8acf-4668-96ff-61c7c16fb10d-kube-api-access-dtfrq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:06:20.739612 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.739606 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700cdeee-8acf-4668-96ff-61c7c16fb10d-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:06:20.739804 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:20.739618 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700cdeee-8acf-4668-96ff-61c7c16fb10d-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:06:21.004533 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.004498 2582 generic.go:356] "Generic (PLEG): container finished" podID="700cdeee-8acf-4668-96ff-61c7c16fb10d" containerID="089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86" exitCode=0 Sep 17 02:06:21.004704 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.004544 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" Sep 17 02:06:21.004704 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.004588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" event={"ID":"700cdeee-8acf-4668-96ff-61c7c16fb10d","Type":"ContainerDied","Data":"089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86"} Sep 17 02:06:21.004704 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.004632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d" event={"ID":"700cdeee-8acf-4668-96ff-61c7c16fb10d","Type":"ContainerDied","Data":"e6b2fdaa4818db3ac50fa32fbd675843cc1025bdeda5603b70acfbb6aa9599d0"} Sep 17 02:06:21.004704 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.004655 2582 scope.go:117] "RemoveContainer" containerID="089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86" Sep 17 02:06:21.017120 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.017096 2582 scope.go:117] "RemoveContainer" containerID="089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86" Sep 17 02:06:21.017517 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:06:21.017488 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86\": container with ID starting with 089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86 not found: ID does not exist" containerID="089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86" Sep 17 02:06:21.017628 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.017527 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86"} err="failed to get container status \"089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86\": rpc error: code = NotFound desc = could not find container \"089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86\": container with ID starting with 089d4e0438b1624be574c62de87cdcda445b7eaa199c4a3a549d6fc92185be86 not found: ID does not exist" Sep 17 02:06:21.027449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.027425 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d"] Sep 17 02:06:21.031018 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.030995 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/auth-test-model-f03337-kserve-cb7ffbc55-jlk9d"] Sep 17 02:06:21.679573 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:21.679532 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="700cdeee-8acf-4668-96ff-61c7c16fb10d" path="/var/lib/kubelet/pods/700cdeee-8acf-4668-96ff-61c7c16fb10d/volumes" Sep 17 02:06:25.523120 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:25.523089 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:06:26.319546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:26.319512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:27.651543 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:27.651506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:28.424798 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:28.424754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:29.238783 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:29.238742 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:29.483304 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:29.483267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:29.483304 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:29.483309 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:29.495573 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:29.495510 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:29.937869 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:29.937768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:30.071976 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:30.071949 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:06:30.695265 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:30.695226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:31.449225 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:31.449186 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:32.351969 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:32.351933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:32.822377 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:32.822336 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:33.160283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:33.160181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:33.588829 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:33.588791 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:35.702502 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:35.702462 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:35.862113 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:35.862081 2582 scope.go:117] "RemoveContainer" containerID="82b99552d8d629631df07cef4203bf1bf446d7f86b954d86bd3853020b5a5044" Sep 17 02:06:35.870695 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:35.870669 2582 scope.go:117] "RemoveContainer" containerID="c5ceff9dc48c0f9e5d70c1d5b953b374a42bb1c6ccf9a8d7e5ea26f1b683e1d9" Sep 17 02:06:35.879232 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:35.879201 2582 scope.go:117] "RemoveContainer" containerID="7fcc71f83af3bb55bbeb767b6f5865d3fd6613a2e7e0018abfd58f59e76f75fe" Sep 17 02:06:35.887276 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:35.887248 2582 scope.go:117] "RemoveContainer" containerID="7ecb00685c4e1ca4c751e7ceb61ea0ea070124ea4f1c5020233f87208d2937ee" Sep 17 02:06:36.386129 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:36.386091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:36.467240 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:36.467185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:36.620735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:36.620695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:37.424331 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:37.424293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:38.224006 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:38.223961 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:38.970245 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:38.970192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:39.501073 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:39.501039 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:40.458532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:40.458500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:40.559071 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:40.559038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:40.683400 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:40.683365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:40.922200 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:40.922159 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:41.019501 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.019467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:41.048788 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.048759 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd"] Sep 17 02:06:41.049188 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.049174 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="700cdeee-8acf-4668-96ff-61c7c16fb10d" containerName="main" Sep 17 02:06:41.049271 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.049189 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="700cdeee-8acf-4668-96ff-61c7c16fb10d" containerName="main" Sep 17 02:06:41.049271 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.049256 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="700cdeee-8acf-4668-96ff-61c7c16fb10d" containerName="main" Sep 17 02:06:41.052462 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.052437 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.054909 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.054885 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-320e88\"" Sep 17 02:06:41.055057 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.054952 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-320e88-dockercfg-526w7\"" Sep 17 02:06:41.061090 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.060898 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd"] Sep 17 02:06:41.124654 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.124623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8196c5-45de-4883-89ea-c51e8246f226-plugins-config-volume\") pod \"payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.124866 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.124661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlr6j\" (UniqueName: \"kubernetes.io/projected/6d8196c5-45de-4883-89ea-c51e8246f226-kube-api-access-xlr6j\") pod \"payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.226098 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.226062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlr6j\" (UniqueName: \"kubernetes.io/projected/6d8196c5-45de-4883-89ea-c51e8246f226-kube-api-access-xlr6j\") pod \"payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.226309 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.226180 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8196c5-45de-4883-89ea-c51e8246f226-plugins-config-volume\") pod \"payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.226905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.226887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8196c5-45de-4883-89ea-c51e8246f226-plugins-config-volume\") pod \"payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.233643 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.233624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlr6j\" (UniqueName: \"kubernetes.io/projected/6d8196c5-45de-4883-89ea-c51e8246f226-kube-api-access-xlr6j\") pod \"payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.333575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.333546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:41.364346 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.364328 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:41.638837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:41.638811 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd"] Sep 17 02:06:41.640342 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:06:41.640313 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8196c5_45de_4883_89ea_c51e8246f226.slice/crio-20f4ff4cbcccebcd37ab14898bb81c53475733e2457538d2ad1adebd073e17c2 WatchSource:0}: Error finding container 20f4ff4cbcccebcd37ab14898bb81c53475733e2457538d2ad1adebd073e17c2: Status 404 returned error can't find the container with id 20f4ff4cbcccebcd37ab14898bb81c53475733e2457538d2ad1adebd073e17c2 Sep 17 02:06:42.106278 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:42.106245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:42.129545 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:42.129514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" event={"ID":"6d8196c5-45de-4883-89ea-c51e8246f226","Type":"ContainerStarted","Data":"ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b"} Sep 17 02:06:42.129723 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:42.129565 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" event={"ID":"6d8196c5-45de-4883-89ea-c51e8246f226","Type":"ContainerStarted","Data":"20f4ff4cbcccebcd37ab14898bb81c53475733e2457538d2ad1adebd073e17c2"} Sep 17 02:06:42.129723 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:42.129593 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:42.145250 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:42.145192 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" podStartSLOduration=1.145176589 podStartE2EDuration="1.145176589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:06:42.143877801 +0000 UTC m=+1746.978930135" watchObservedRunningTime="2026-09-17 02:06:42.145176589 +0000 UTC m=+1746.980228928" Sep 17 02:06:44.947416 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:44.947380 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:46.403480 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:46.403440 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:47.973085 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:47.973044 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:06:53.137745 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:53.137712 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:06:57.277941 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.277911 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5"] Sep 17 02:06:57.281871 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.281846 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.285450 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.285424 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"sub-test-model-f03337-kserve-self-signed-certs\"" Sep 17 02:06:57.285634 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.285455 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"default-dockercfg-xb9xt\"" Sep 17 02:06:57.285634 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.285491 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"kube-root-ca.crt\"" Sep 17 02:06:57.285877 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.285795 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"openshift-service-ca.crt\"" Sep 17 02:06:57.289491 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.289468 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5"] Sep 17 02:06:57.309900 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.309876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:06:57.378920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.378894 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-model-cache\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.379041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.378936 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-dshm\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.379041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.378959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455253b8-ce4c-41a7-b322-f25958344211-tls-certs\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.379041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.379008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-home\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.379233 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.379107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jvq6\" (UniqueName: \"kubernetes.io/projected/455253b8-ce4c-41a7-b322-f25958344211-kube-api-access-7jvq6\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.379233 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.379182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-tmp-dir\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.479999 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.479974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-tmp-dir\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480105 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-model-cache\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480174 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480141 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-dshm\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480258 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480188 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455253b8-ce4c-41a7-b322-f25958344211-tls-certs\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480335 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-home\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jvq6\" (UniqueName: \"kubernetes.io/projected/455253b8-ce4c-41a7-b322-f25958344211-kube-api-access-7jvq6\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-model-cache\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480408 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-tmp-dir\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.480602 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.480583 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-home\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.482261 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.482241 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-dshm\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.482772 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.482752 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455253b8-ce4c-41a7-b322-f25958344211-tls-certs\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.488035 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.488011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jvq6\" (UniqueName: \"kubernetes.io/projected/455253b8-ce4c-41a7-b322-f25958344211-kube-api-access-7jvq6\") pod \"sub-test-model-f03337-kserve-857dc7675-jzjx5\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.592945 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.592879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:06:57.864155 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:57.864134 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5"] Sep 17 02:06:57.866073 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:06:57.866040 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod455253b8_ce4c_41a7_b322_f25958344211.slice/crio-307beffc8909f39dcae3431280ab03174e80fdf185e4e419c205a36133c0f0d8 WatchSource:0}: Error finding container 307beffc8909f39dcae3431280ab03174e80fdf185e4e419c205a36133c0f0d8: Status 404 returned error can't find the container with id 307beffc8909f39dcae3431280ab03174e80fdf185e4e419c205a36133c0f0d8 Sep 17 02:06:58.216721 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:58.216691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" event={"ID":"455253b8-ce4c-41a7-b322-f25958344211","Type":"ContainerStarted","Data":"2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5"} Sep 17 02:06:58.216914 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:58.216727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" event={"ID":"455253b8-ce4c-41a7-b322-f25958344211","Type":"ContainerStarted","Data":"307beffc8909f39dcae3431280ab03174e80fdf185e4e419c205a36133c0f0d8"} Sep 17 02:06:58.232703 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:06:58.232655 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" podStartSLOduration=1.232640835 podStartE2EDuration="1.232640835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:06:58.231155887 +0000 UTC m=+1763.066208219" watchObservedRunningTime="2026-09-17 02:06:58.232640835 +0000 UTC m=+1763.067693167" Sep 17 02:07:01.774847 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:01.774813 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:07:02.657725 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:02.657681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:03.572977 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:03.572931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:04.363933 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:04.363904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:05.054006 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:05.053968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:05.753022 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:05.752990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:06.293607 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:06.293575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:06.555491 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:06.555403 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:07.047828 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:07.047789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:07.593921 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:07.593889 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:07:07.594373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:07.593970 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:07:07.606256 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:07.606235 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:07:08.282514 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:08.282486 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:07:09.639997 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.639969 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh"] Sep 17 02:07:09.643401 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.643384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:09.645850 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.645825 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:07:09.645973 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.645888 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:07:09.649793 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.649768 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh"] Sep 17 02:07:09.679799 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.679774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec9715-5387-4152-bbcb-79c6bab0c13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh\" (UID: \"c3ec9715-5387-4152-bbcb-79c6bab0c13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:09.780324 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.780305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec9715-5387-4152-bbcb-79c6bab0c13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh\" (UID: \"c3ec9715-5387-4152-bbcb-79c6bab0c13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:09.780881 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.780852 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec9715-5387-4152-bbcb-79c6bab0c13a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh\" (UID: \"c3ec9715-5387-4152-bbcb-79c6bab0c13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:09.938439 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.938362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:09.955836 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:09.955817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:10.227017 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:10.226988 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh"] Sep 17 02:07:10.228165 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:07:10.228139 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ec9715_5387_4152_bbcb_79c6bab0c13a.slice/crio-3c0ec435e487ddaa32223a4f5bffbc9ebfa091b50e9d97fafa392e4d447240f8 WatchSource:0}: Error finding container 3c0ec435e487ddaa32223a4f5bffbc9ebfa091b50e9d97fafa392e4d447240f8: Status 404 returned error can't find the container with id 3c0ec435e487ddaa32223a4f5bffbc9ebfa091b50e9d97fafa392e4d447240f8 Sep 17 02:07:10.286020 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:10.285994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" event={"ID":"c3ec9715-5387-4152-bbcb-79c6bab0c13a","Type":"ContainerStarted","Data":"3c0ec435e487ddaa32223a4f5bffbc9ebfa091b50e9d97fafa392e4d447240f8"} Sep 17 02:07:10.661003 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:10.660920 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:11.293333 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:11.293297 2582 generic.go:356] "Generic (PLEG): container finished" podID="c3ec9715-5387-4152-bbcb-79c6bab0c13a" containerID="c7033da96860bbae2fbd4c6e61330db54bdfee51d6b3d471ad7224edac0af73c" exitCode=0 Sep 17 02:07:11.293528 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:11.293383 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" event={"ID":"c3ec9715-5387-4152-bbcb-79c6bab0c13a","Type":"ContainerDied","Data":"c7033da96860bbae2fbd4c6e61330db54bdfee51d6b3d471ad7224edac0af73c"} Sep 17 02:07:11.343854 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:11.343819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:12.089291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:12.089251 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:12.430001 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:12.429976 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:12.501482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:12.501449 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec9715-5387-4152-bbcb-79c6bab0c13a-maas-api-service-ca\") pod \"c3ec9715-5387-4152-bbcb-79c6bab0c13a\" (UID: \"c3ec9715-5387-4152-bbcb-79c6bab0c13a\") " Sep 17 02:07:12.501837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:12.501814 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ec9715-5387-4152-bbcb-79c6bab0c13a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3ec9715-5387-4152-bbcb-79c6bab0c13a" (UID: "c3ec9715-5387-4152-bbcb-79c6bab0c13a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:07:12.602550 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:12.602462 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec9715-5387-4152-bbcb-79c6bab0c13a-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:12.770155 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:12.770118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:13.305456 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:13.305415 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" event={"ID":"c3ec9715-5387-4152-bbcb-79c6bab0c13a","Type":"ContainerDied","Data":"3c0ec435e487ddaa32223a4f5bffbc9ebfa091b50e9d97fafa392e4d447240f8"} Sep 17 02:07:13.305899 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:13.305477 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c0ec435e487ddaa32223a4f5bffbc9ebfa091b50e9d97fafa392e4d447240f8" Sep 17 02:07:13.305899 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:13.305432 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh" Sep 17 02:07:13.554041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:13.553999 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:14.260531 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:14.260492 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:15.037154 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:15.037115 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:15.683100 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:15.683069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:19.819884 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:19.819847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:25.394536 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.394501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:25.457514 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.457477 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:25.556090 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.556051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:25.573612 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.573583 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:07:25.588713 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.588687 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh"] Sep 17 02:07:25.591880 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.591861 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-11a2e3-2qvhh"] Sep 17 02:07:25.618573 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.618539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:25.679695 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.679643 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ec9715-5387-4152-bbcb-79c6bab0c13a" path="/var/lib/kubelet/pods/c3ec9715-5387-4152-bbcb-79c6bab0c13a/volumes" Sep 17 02:07:25.957111 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:25.957026 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:26.050405 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:26.050361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:26.785283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:26.785225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:27.638912 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:27.638876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:28.324812 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:28.324771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:28.996333 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:28.996302 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:29.635125 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:29.635089 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:31.968353 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:31.968320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:32.070838 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.070808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:32.139283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.139253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:32.457346 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.457270 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr"] Sep 17 02:07:32.457710 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.457696 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec9715-5387-4152-bbcb-79c6bab0c13a" containerName="revoke-keys" Sep 17 02:07:32.457762 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.457711 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec9715-5387-4152-bbcb-79c6bab0c13a" containerName="revoke-keys" Sep 17 02:07:32.457795 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.457779 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec9715-5387-4152-bbcb-79c6bab0c13a" containerName="revoke-keys" Sep 17 02:07:32.461261 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.461246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:32.463458 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.463436 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:07:32.463593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.463438 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:07:32.467416 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.467392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr"] Sep 17 02:07:32.584605 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.584572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79353f1-613c-4a79-b2eb-a573e8943676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-320e88-656gr\" (UID: \"e79353f1-613c-4a79-b2eb-a573e8943676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:32.685724 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.685699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79353f1-613c-4a79-b2eb-a573e8943676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-320e88-656gr\" (UID: \"e79353f1-613c-4a79-b2eb-a573e8943676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:32.686329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.686282 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79353f1-613c-4a79-b2eb-a573e8943676-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-320e88-656gr\" (UID: \"e79353f1-613c-4a79-b2eb-a573e8943676\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:32.718329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.718303 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:32.772512 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:32.772486 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:33.046508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:33.046399 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr"] Sep 17 02:07:33.048760 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:07:33.048733 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79353f1_613c_4a79_b2eb_a573e8943676.slice/crio-1e1a019f48225eeeb5a8f09c2918a940c71c37876aff274b2b39f7d20b786744 WatchSource:0}: Error finding container 1e1a019f48225eeeb5a8f09c2918a940c71c37876aff274b2b39f7d20b786744: Status 404 returned error can't find the container with id 1e1a019f48225eeeb5a8f09c2918a940c71c37876aff274b2b39f7d20b786744 Sep 17 02:07:33.419587 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:33.419514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" event={"ID":"e79353f1-613c-4a79-b2eb-a573e8943676","Type":"ContainerStarted","Data":"08b41eb384e16071a1488ff7e872b1d857768c118c55cb4ae338fcfd8250cdb5"} Sep 17 02:07:33.419587 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:33.419576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" event={"ID":"e79353f1-613c-4a79-b2eb-a573e8943676","Type":"ContainerStarted","Data":"1e1a019f48225eeeb5a8f09c2918a940c71c37876aff274b2b39f7d20b786744"} Sep 17 02:07:33.435119 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:33.435077 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" podStartSLOduration=1.435063346 podStartE2EDuration="1.435063346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:07:33.433260424 +0000 UTC m=+1798.268312758" watchObservedRunningTime="2026-09-17 02:07:33.435063346 +0000 UTC m=+1798.270115679" Sep 17 02:07:33.548189 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:33.548155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:34.227695 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:34.227643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:34.426510 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:34.426439 2582 generic.go:356] "Generic (PLEG): container finished" podID="e79353f1-613c-4a79-b2eb-a573e8943676" containerID="08b41eb384e16071a1488ff7e872b1d857768c118c55cb4ae338fcfd8250cdb5" exitCode=0 Sep 17 02:07:34.426644 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:34.426525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" event={"ID":"e79353f1-613c-4a79-b2eb-a573e8943676","Type":"ContainerDied","Data":"08b41eb384e16071a1488ff7e872b1d857768c118c55cb4ae338fcfd8250cdb5"} Sep 17 02:07:34.888606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:34.888575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:35.551892 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:35.551861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:35.560447 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:35.560427 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:35.712738 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:35.712714 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79353f1-613c-4a79-b2eb-a573e8943676-maas-api-service-ca\") pod \"e79353f1-613c-4a79-b2eb-a573e8943676\" (UID: \"e79353f1-613c-4a79-b2eb-a573e8943676\") " Sep 17 02:07:35.713152 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:35.713127 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79353f1-613c-4a79-b2eb-a573e8943676-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e79353f1-613c-4a79-b2eb-a573e8943676" (UID: "e79353f1-613c-4a79-b2eb-a573e8943676"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:07:35.813457 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:35.813424 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79353f1-613c-4a79-b2eb-a573e8943676-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:35.923021 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:35.922995 2582 scope.go:117] "RemoveContainer" containerID="33d1fee540462bcbb019c55188101a465b8a1ede83767f47f9fb265e0a039d3d" Sep 17 02:07:36.220728 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:36.220694 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:36.437807 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:36.437769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" event={"ID":"e79353f1-613c-4a79-b2eb-a573e8943676","Type":"ContainerDied","Data":"1e1a019f48225eeeb5a8f09c2918a940c71c37876aff274b2b39f7d20b786744"} Sep 17 02:07:36.437807 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:36.437801 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr" Sep 17 02:07:36.437994 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:36.437861 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e1a019f48225eeeb5a8f09c2918a940c71c37876aff274b2b39f7d20b786744" Sep 17 02:07:36.881380 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:36.881350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:42.562179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:42.562142 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg"] Sep 17 02:07:42.590821 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:42.590787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:43.471156 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:43.471112 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5"] Sep 17 02:07:43.496840 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:43.496810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:45.567526 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:45.567489 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:07:47.998594 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:47.998559 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:48.034170 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.034135 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd"] Sep 17 02:07:48.034512 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.034480 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" podUID="6d8196c5-45de-4883-89ea-c51e8246f226" containerName="payload-processing" containerID="cri-o://ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b" gracePeriod=30 Sep 17 02:07:48.101364 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.101322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:48.199346 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.199311 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr"] Sep 17 02:07:48.200600 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.200557 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:48.201946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.201919 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-320e88-656gr"] Sep 17 02:07:48.293525 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.293501 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:07:48.298826 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.298797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:48.439757 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.439727 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8196c5-45de-4883-89ea-c51e8246f226-plugins-config-volume\") pod \"6d8196c5-45de-4883-89ea-c51e8246f226\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " Sep 17 02:07:48.439960 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.439825 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlr6j\" (UniqueName: \"kubernetes.io/projected/6d8196c5-45de-4883-89ea-c51e8246f226-kube-api-access-xlr6j\") pod \"6d8196c5-45de-4883-89ea-c51e8246f226\" (UID: \"6d8196c5-45de-4883-89ea-c51e8246f226\") " Sep 17 02:07:48.440135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.440113 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8196c5-45de-4883-89ea-c51e8246f226-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d8196c5-45de-4883-89ea-c51e8246f226" (UID: "6d8196c5-45de-4883-89ea-c51e8246f226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:07:48.441905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.441879 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8196c5-45de-4883-89ea-c51e8246f226-kube-api-access-xlr6j" (OuterVolumeSpecName: "kube-api-access-xlr6j") pod "6d8196c5-45de-4883-89ea-c51e8246f226" (UID: "6d8196c5-45de-4883-89ea-c51e8246f226"). InnerVolumeSpecName "kube-api-access-xlr6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:48.511506 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.511476 2582 generic.go:356] "Generic (PLEG): container finished" podID="6d8196c5-45de-4883-89ea-c51e8246f226" containerID="ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b" exitCode=0 Sep 17 02:07:48.511669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.511515 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" Sep 17 02:07:48.511669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.511540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" event={"ID":"6d8196c5-45de-4883-89ea-c51e8246f226","Type":"ContainerDied","Data":"ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b"} Sep 17 02:07:48.511669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.511606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd" event={"ID":"6d8196c5-45de-4883-89ea-c51e8246f226","Type":"ContainerDied","Data":"20f4ff4cbcccebcd37ab14898bb81c53475733e2457538d2ad1adebd073e17c2"} Sep 17 02:07:48.511669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.511627 2582 scope.go:117] "RemoveContainer" containerID="ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b" Sep 17 02:07:48.522040 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.521986 2582 scope.go:117] "RemoveContainer" containerID="ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b" Sep 17 02:07:48.522441 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:07:48.522404 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b\": container with ID starting with ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b not found: ID does not exist" containerID="ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b" Sep 17 02:07:48.522544 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.522447 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b"} err="failed to get container status \"ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b\": rpc error: code = NotFound desc = could not find container \"ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b\": container with ID starting with ae45d3ecab82d9330fc8e1fef2ce47b0d580aa10789a5a2eb1687d549ba03f4b not found: ID does not exist" Sep 17 02:07:48.538122 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.538095 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd"] Sep 17 02:07:48.541462 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.541440 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8196c5-45de-4883-89ea-c51e8246f226-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:48.541462 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.541462 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlr6j\" (UniqueName: \"kubernetes.io/projected/6d8196c5-45de-4883-89ea-c51e8246f226-kube-api-access-xlr6j\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:48.543575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.543553 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-320e88-64c54db8c7-zv4fd"] Sep 17 02:07:48.567873 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.567848 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:48.669279 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:48.669246 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:49.349672 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:49.349630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:49.679012 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:49.678942 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8196c5-45de-4883-89ea-c51e8246f226" path="/var/lib/kubelet/pods/6d8196c5-45de-4883-89ea-c51e8246f226/volumes" Sep 17 02:07:49.679329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:49.679312 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79353f1-613c-4a79-b2eb-a573e8943676" path="/var/lib/kubelet/pods/e79353f1-613c-4a79-b2eb-a573e8943676/volumes" Sep 17 02:07:49.996180 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:49.996145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:50.755426 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:50.755379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:51.367665 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:51.367632 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:51.647964 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:51.647869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:54.333730 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.333691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:54.396861 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.396822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:54.599460 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.599375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:54.970997 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.970971 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv"] Sep 17 02:07:54.971395 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.971382 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79353f1-613c-4a79-b2eb-a573e8943676" containerName="revoke-keys" Sep 17 02:07:54.971449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.971396 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79353f1-613c-4a79-b2eb-a573e8943676" containerName="revoke-keys" Sep 17 02:07:54.971449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.971413 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8196c5-45de-4883-89ea-c51e8246f226" containerName="payload-processing" Sep 17 02:07:54.971449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.971419 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8196c5-45de-4883-89ea-c51e8246f226" containerName="payload-processing" Sep 17 02:07:54.971545 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.971481 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8196c5-45de-4883-89ea-c51e8246f226" containerName="payload-processing" Sep 17 02:07:54.971545 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.971491 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79353f1-613c-4a79-b2eb-a573e8943676" containerName="revoke-keys" Sep 17 02:07:54.975412 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.975394 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:54.977571 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.977548 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:07:54.977687 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.977607 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:07:54.981355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.981333 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv"] Sep 17 02:07:54.995686 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:54.995657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8db7fb-8bff-42f5-b200-7d71c95bacf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv\" (UID: \"7c8db7fb-8bff-42f5-b200-7d71c95bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:55.096467 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:55.096436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8db7fb-8bff-42f5-b200-7d71c95bacf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv\" (UID: \"7c8db7fb-8bff-42f5-b200-7d71c95bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:55.097069 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:55.097043 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8db7fb-8bff-42f5-b200-7d71c95bacf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv\" (UID: \"7c8db7fb-8bff-42f5-b200-7d71c95bacf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:55.122296 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:55.122266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:55.286857 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:55.286785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:55.556087 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:55.556062 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv"] Sep 17 02:07:55.557924 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:07:55.557888 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8db7fb_8bff_42f5_b200_7d71c95bacf0.slice/crio-159813d9f2ddb1b68075d9b6d87cf929822e69f17d640762440115aaf496d1ba WatchSource:0}: Error finding container 159813d9f2ddb1b68075d9b6d87cf929822e69f17d640762440115aaf496d1ba: Status 404 returned error can't find the container with id 159813d9f2ddb1b68075d9b6d87cf929822e69f17d640762440115aaf496d1ba Sep 17 02:07:55.861065 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:55.860996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:56.555695 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:56.555616 2582 generic.go:356] "Generic (PLEG): container finished" podID="7c8db7fb-8bff-42f5-b200-7d71c95bacf0" containerID="210f53f1a18697b63013e0012f06119188ce1f0a29afb85efd1a74e5c7150783" exitCode=0 Sep 17 02:07:56.555832 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:56.555702 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" event={"ID":"7c8db7fb-8bff-42f5-b200-7d71c95bacf0","Type":"ContainerDied","Data":"210f53f1a18697b63013e0012f06119188ce1f0a29afb85efd1a74e5c7150783"} Sep 17 02:07:56.555832 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:56.555754 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" event={"ID":"7c8db7fb-8bff-42f5-b200-7d71c95bacf0","Type":"ContainerStarted","Data":"159813d9f2ddb1b68075d9b6d87cf929822e69f17d640762440115aaf496d1ba"} Sep 17 02:07:56.742162 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:56.742126 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:57.303939 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.303903 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:57.582407 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.582369 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" podUID="760ec0bd-4f8f-499b-8857-d303e689745a" containerName="main" containerID="cri-o://7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e" gracePeriod=45 Sep 17 02:07:57.708135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.708112 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:57.821965 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.821882 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8db7fb-8bff-42f5-b200-7d71c95bacf0-maas-api-service-ca\") pod \"7c8db7fb-8bff-42f5-b200-7d71c95bacf0\" (UID: \"7c8db7fb-8bff-42f5-b200-7d71c95bacf0\") " Sep 17 02:07:57.822450 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.822411 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8db7fb-8bff-42f5-b200-7d71c95bacf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c8db7fb-8bff-42f5-b200-7d71c95bacf0" (UID: "7c8db7fb-8bff-42f5-b200-7d71c95bacf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:07:57.850016 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.849993 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:07:57.923352 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923317 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-dshm\") pod \"760ec0bd-4f8f-499b-8857-d303e689745a\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " Sep 17 02:07:57.923508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923368 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760ec0bd-4f8f-499b-8857-d303e689745a-tls-certs\") pod \"760ec0bd-4f8f-499b-8857-d303e689745a\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " Sep 17 02:07:57.923508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923405 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dxsq\" (UniqueName: \"kubernetes.io/projected/760ec0bd-4f8f-499b-8857-d303e689745a-kube-api-access-9dxsq\") pod \"760ec0bd-4f8f-499b-8857-d303e689745a\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " Sep 17 02:07:57.923508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923487 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-model-cache\") pod \"760ec0bd-4f8f-499b-8857-d303e689745a\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " Sep 17 02:07:57.923676 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923580 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-tmp-dir\") pod \"760ec0bd-4f8f-499b-8857-d303e689745a\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " Sep 17 02:07:57.923676 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923613 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-home\") pod \"760ec0bd-4f8f-499b-8857-d303e689745a\" (UID: \"760ec0bd-4f8f-499b-8857-d303e689745a\") " Sep 17 02:07:57.923813 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923753 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-model-cache" (OuterVolumeSpecName: "model-cache") pod "760ec0bd-4f8f-499b-8857-d303e689745a" (UID: "760ec0bd-4f8f-499b-8857-d303e689745a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:57.924102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923851 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "760ec0bd-4f8f-499b-8857-d303e689745a" (UID: "760ec0bd-4f8f-499b-8857-d303e689745a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:57.924102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.923883 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-home" (OuterVolumeSpecName: "home") pod "760ec0bd-4f8f-499b-8857-d303e689745a" (UID: "760ec0bd-4f8f-499b-8857-d303e689745a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:57.924102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.924032 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8db7fb-8bff-42f5-b200-7d71c95bacf0-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:57.924102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.924057 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:57.924102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.924070 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:57.924102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.924083 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:57.925599 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.925579 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-dshm" (OuterVolumeSpecName: "dshm") pod "760ec0bd-4f8f-499b-8857-d303e689745a" (UID: "760ec0bd-4f8f-499b-8857-d303e689745a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:57.925699 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.925649 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760ec0bd-4f8f-499b-8857-d303e689745a-kube-api-access-9dxsq" (OuterVolumeSpecName: "kube-api-access-9dxsq") pod "760ec0bd-4f8f-499b-8857-d303e689745a" (UID: "760ec0bd-4f8f-499b-8857-d303e689745a"). InnerVolumeSpecName "kube-api-access-9dxsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:57.925699 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:57.925666 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760ec0bd-4f8f-499b-8857-d303e689745a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "760ec0bd-4f8f-499b-8857-d303e689745a" (UID: "760ec0bd-4f8f-499b-8857-d303e689745a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:07:58.025527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.025501 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760ec0bd-4f8f-499b-8857-d303e689745a-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.025527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.025524 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760ec0bd-4f8f-499b-8857-d303e689745a-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.025678 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.025534 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxsq\" (UniqueName: \"kubernetes.io/projected/760ec0bd-4f8f-499b-8857-d303e689745a-kube-api-access-9dxsq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.041573 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.041545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:58.490721 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.490676 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" podUID="455253b8-ce4c-41a7-b322-f25958344211" containerName="main" containerID="cri-o://2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5" gracePeriod=45 Sep 17 02:07:58.570345 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.570306 2582 generic.go:356] "Generic (PLEG): container finished" podID="760ec0bd-4f8f-499b-8857-d303e689745a" containerID="7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e" exitCode=0 Sep 17 02:07:58.570482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.570353 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" Sep 17 02:07:58.570482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.570393 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" event={"ID":"760ec0bd-4f8f-499b-8857-d303e689745a","Type":"ContainerDied","Data":"7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e"} Sep 17 02:07:58.570482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.570444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg" event={"ID":"760ec0bd-4f8f-499b-8857-d303e689745a","Type":"ContainerDied","Data":"012854fa412a77d024f9f1e3d7bd5943d72f8f0698d456c0b962313402637d01"} Sep 17 02:07:58.570482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.570472 2582 scope.go:117] "RemoveContainer" containerID="7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e" Sep 17 02:07:58.571827 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.571812 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" Sep 17 02:07:58.571928 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.571810 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv" event={"ID":"7c8db7fb-8bff-42f5-b200-7d71c95bacf0","Type":"ContainerDied","Data":"159813d9f2ddb1b68075d9b6d87cf929822e69f17d640762440115aaf496d1ba"} Sep 17 02:07:58.571985 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.571950 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159813d9f2ddb1b68075d9b6d87cf929822e69f17d640762440115aaf496d1ba" Sep 17 02:07:58.593805 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.593778 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg"] Sep 17 02:07:58.601887 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.601856 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-67a23a/sub-test-model-67a23a-kserve-7f76f8d86c-784mg"] Sep 17 02:07:58.607953 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.607936 2582 scope.go:117] "RemoveContainer" containerID="7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e" Sep 17 02:07:58.608251 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:07:58.608228 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e\": container with ID starting with 7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e not found: ID does not exist" containerID="7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e" Sep 17 02:07:58.608329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.608256 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e"} err="failed to get container status \"7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e\": rpc error: code = NotFound desc = could not find container \"7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e\": container with ID starting with 7a0db3b7a1aa8e93fb304df9b3fecd6419b85cd09a7144ff4b4300d589bad47e not found: ID does not exist" Sep 17 02:07:58.702947 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.702911 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:07:58.749041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.748979 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:07:58.832708 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.832671 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-model-cache\") pod \"455253b8-ce4c-41a7-b322-f25958344211\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.832739 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jvq6\" (UniqueName: \"kubernetes.io/projected/455253b8-ce4c-41a7-b322-f25958344211-kube-api-access-7jvq6\") pod \"455253b8-ce4c-41a7-b322-f25958344211\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.832777 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-tmp-dir\") pod \"455253b8-ce4c-41a7-b322-f25958344211\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.832917 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-dshm\") pod \"455253b8-ce4c-41a7-b322-f25958344211\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.832981 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-home\") pod \"455253b8-ce4c-41a7-b322-f25958344211\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.832918 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-model-cache" (OuterVolumeSpecName: "model-cache") pod "455253b8-ce4c-41a7-b322-f25958344211" (UID: "455253b8-ce4c-41a7-b322-f25958344211"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.833039 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "455253b8-ce4c-41a7-b322-f25958344211" (UID: "455253b8-ce4c-41a7-b322-f25958344211"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:58.833183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.833043 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455253b8-ce4c-41a7-b322-f25958344211-tls-certs\") pod \"455253b8-ce4c-41a7-b322-f25958344211\" (UID: \"455253b8-ce4c-41a7-b322-f25958344211\") " Sep 17 02:07:58.833493 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.833281 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-home" (OuterVolumeSpecName: "home") pod "455253b8-ce4c-41a7-b322-f25958344211" (UID: "455253b8-ce4c-41a7-b322-f25958344211"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:58.833493 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.833464 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.833493 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.833480 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.833609 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.833511 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.835088 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.835062 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455253b8-ce4c-41a7-b322-f25958344211-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "455253b8-ce4c-41a7-b322-f25958344211" (UID: "455253b8-ce4c-41a7-b322-f25958344211"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:07:58.835394 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.835378 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-dshm" (OuterVolumeSpecName: "dshm") pod "455253b8-ce4c-41a7-b322-f25958344211" (UID: "455253b8-ce4c-41a7-b322-f25958344211"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:07:58.835474 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.835429 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455253b8-ce4c-41a7-b322-f25958344211-kube-api-access-7jvq6" (OuterVolumeSpecName: "kube-api-access-7jvq6") pod "455253b8-ce4c-41a7-b322-f25958344211" (UID: "455253b8-ce4c-41a7-b322-f25958344211"). InnerVolumeSpecName "kube-api-access-7jvq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:07:58.934593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.934558 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jvq6\" (UniqueName: \"kubernetes.io/projected/455253b8-ce4c-41a7-b322-f25958344211-kube-api-access-7jvq6\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.934593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.934590 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/455253b8-ce4c-41a7-b322-f25958344211-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:58.934593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:58.934602 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/455253b8-ce4c-41a7-b322-f25958344211-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:07:59.579294 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.579261 2582 generic.go:356] "Generic (PLEG): container finished" podID="455253b8-ce4c-41a7-b322-f25958344211" containerID="2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5" exitCode=0 Sep 17 02:07:59.579486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.579304 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" Sep 17 02:07:59.579486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.579346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" event={"ID":"455253b8-ce4c-41a7-b322-f25958344211","Type":"ContainerDied","Data":"2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5"} Sep 17 02:07:59.579486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.579390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5" event={"ID":"455253b8-ce4c-41a7-b322-f25958344211","Type":"ContainerDied","Data":"307beffc8909f39dcae3431280ab03174e80fdf185e4e419c205a36133c0f0d8"} Sep 17 02:07:59.579486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.579411 2582 scope.go:117] "RemoveContainer" containerID="2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5" Sep 17 02:07:59.589604 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.589588 2582 scope.go:117] "RemoveContainer" containerID="2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5" Sep 17 02:07:59.589861 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:07:59.589843 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5\": container with ID starting with 2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5 not found: ID does not exist" containerID="2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5" Sep 17 02:07:59.589914 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.589870 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5"} err="failed to get container status \"2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5\": rpc error: code = NotFound desc = could not find container \"2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5\": container with ID starting with 2f5d8838ec87384fd6c78200a818ba238279a591d653fa03c1e37a01429c8cc5 not found: ID does not exist" Sep 17 02:07:59.599401 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.599378 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5"] Sep 17 02:07:59.603059 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.603039 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/sub-test-model-f03337-kserve-857dc7675-jzjx5"] Sep 17 02:07:59.680921 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.680890 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455253b8-ce4c-41a7-b322-f25958344211" path="/var/lib/kubelet/pods/455253b8-ce4c-41a7-b322-f25958344211/volumes" Sep 17 02:07:59.681322 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:07:59.681308 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760ec0bd-4f8f-499b-8857-d303e689745a" path="/var/lib/kubelet/pods/760ec0bd-4f8f-499b-8857-d303e689745a/volumes" Sep 17 02:08:03.872942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:03.872836 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:08:05.468173 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.468134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:05.504089 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.504056 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh"] Sep 17 02:08:05.504351 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.504324 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" podUID="9f805c03-4256-42c9-81c5-f83dd6ba4bbf" containerName="payload-pre-processing" containerID="cri-o://c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207" gracePeriod=30 Sep 17 02:08:05.567850 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.567815 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:05.663868 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.663825 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv"] Sep 17 02:08:05.666598 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.666567 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c5da3b-4m7dv"] Sep 17 02:08:05.666756 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.666669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:05.680010 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.679979 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8db7fb-8bff-42f5-b200-7d71c95bacf0" path="/var/lib/kubelet/pods/7c8db7fb-8bff-42f5-b200-7d71c95bacf0/volumes" Sep 17 02:08:05.723432 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.723314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:05.770006 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.769977 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:08:05.900303 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.900248 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-plugins-config-volume\") pod \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " Sep 17 02:08:05.900583 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.900404 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj67t\" (UniqueName: \"kubernetes.io/projected/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-kube-api-access-mj67t\") pod \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\" (UID: \"9f805c03-4256-42c9-81c5-f83dd6ba4bbf\") " Sep 17 02:08:05.900851 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.900813 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f805c03-4256-42c9-81c5-f83dd6ba4bbf" (UID: "9f805c03-4256-42c9-81c5-f83dd6ba4bbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:05.902700 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:05.902670 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-kube-api-access-mj67t" (OuterVolumeSpecName: "kube-api-access-mj67t") pod "9f805c03-4256-42c9-81c5-f83dd6ba4bbf" (UID: "9f805c03-4256-42c9-81c5-f83dd6ba4bbf"). InnerVolumeSpecName "kube-api-access-mj67t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:06.001968 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.001878 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj67t\" (UniqueName: \"kubernetes.io/projected/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-kube-api-access-mj67t\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:06.001968 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.001911 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f805c03-4256-42c9-81c5-f83dd6ba4bbf-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:06.030923 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.030888 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:06.108488 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.108448 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:06.622355 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.622315 2582 generic.go:356] "Generic (PLEG): container finished" podID="9f805c03-4256-42c9-81c5-f83dd6ba4bbf" containerID="c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207" exitCode=0 Sep 17 02:08:06.622952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.622363 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" Sep 17 02:08:06.622952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.622399 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" event={"ID":"9f805c03-4256-42c9-81c5-f83dd6ba4bbf","Type":"ContainerDied","Data":"c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207"} Sep 17 02:08:06.622952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.622454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh" event={"ID":"9f805c03-4256-42c9-81c5-f83dd6ba4bbf","Type":"ContainerDied","Data":"03990abc7d7d2b982000048ea6532a2cd416f92c205d1b6d522b84190fb3b962"} Sep 17 02:08:06.622952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.622481 2582 scope.go:117] "RemoveContainer" containerID="c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207" Sep 17 02:08:06.632139 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.632120 2582 scope.go:117] "RemoveContainer" containerID="c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207" Sep 17 02:08:06.632422 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:08:06.632401 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207\": container with ID starting with c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207 not found: ID does not exist" containerID="c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207" Sep 17 02:08:06.632513 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.632429 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207"} err="failed to get container status \"c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207\": rpc error: code = NotFound desc = could not find container \"c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207\": container with ID starting with c321c6faf153f87fd3999ff76e46d59e1dab48b800bf4cd79d3789c680626207 not found: ID does not exist" Sep 17 02:08:06.644145 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.644118 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh"] Sep 17 02:08:06.647842 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.647817 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c5da3b-654d966c78-w5vlh"] Sep 17 02:08:06.832069 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:06.832033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:07.393735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:07.393699 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:07.678990 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:07.678912 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f805c03-4256-42c9-81c5-f83dd6ba4bbf" path="/var/lib/kubelet/pods/9f805c03-4256-42c9-81c5-f83dd6ba4bbf/volumes" Sep 17 02:08:08.050117 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:08.050085 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:12.302704 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.302667 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:12.403691 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.403657 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:12.565102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.565017 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth"] Sep 17 02:08:12.565325 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.565298 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" containerID="cri-o://e0a13f6917a514c597635545c4f1ef59a6bacd34005207718097eec4ca86586b" gracePeriod=30 Sep 17 02:08:12.588801 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.588775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:12.948868 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.948792 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9"] Sep 17 02:08:12.949160 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949148 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f805c03-4256-42c9-81c5-f83dd6ba4bbf" containerName="payload-pre-processing" Sep 17 02:08:12.949204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949162 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f805c03-4256-42c9-81c5-f83dd6ba4bbf" containerName="payload-pre-processing" Sep 17 02:08:12.949204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949188 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8db7fb-8bff-42f5-b200-7d71c95bacf0" containerName="revoke-keys" Sep 17 02:08:12.949204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949193 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8db7fb-8bff-42f5-b200-7d71c95bacf0" containerName="revoke-keys" Sep 17 02:08:12.949204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949202 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760ec0bd-4f8f-499b-8857-d303e689745a" containerName="main" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949221 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="760ec0bd-4f8f-499b-8857-d303e689745a" containerName="main" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949228 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455253b8-ce4c-41a7-b322-f25958344211" containerName="main" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949233 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="455253b8-ce4c-41a7-b322-f25958344211" containerName="main" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949290 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f805c03-4256-42c9-81c5-f83dd6ba4bbf" containerName="payload-pre-processing" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949298 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="760ec0bd-4f8f-499b-8857-d303e689745a" containerName="main" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949304 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="455253b8-ce4c-41a7-b322-f25958344211" containerName="main" Sep 17 02:08:12.949356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.949311 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8db7fb-8bff-42f5-b200-7d71c95bacf0" containerName="revoke-keys" Sep 17 02:08:12.952198 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.952182 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:12.954517 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.954492 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:08:12.954647 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.954612 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:08:12.958986 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:12.958961 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9"] Sep 17 02:08:13.069288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.069254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a5e67-629f-49a3-aaaf-0dea8170d80f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9\" (UID: \"0b9a5e67-629f-49a3-aaaf-0dea8170d80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:13.170464 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.170438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a5e67-629f-49a3-aaaf-0dea8170d80f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9\" (UID: \"0b9a5e67-629f-49a3-aaaf-0dea8170d80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:13.170991 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.170972 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a5e67-629f-49a3-aaaf-0dea8170d80f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9\" (UID: \"0b9a5e67-629f-49a3-aaaf-0dea8170d80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:13.238512 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.238488 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:13.263795 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.263771 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:13.538234 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.538193 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9"] Sep 17 02:08:13.540335 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:08:13.540305 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9a5e67_629f_49a3_aaaf_0dea8170d80f.slice/crio-6b0f2aea53ecbf6b6513e9d9887013660064afa5d2caa6b37e01d32f6d40d63f WatchSource:0}: Error finding container 6b0f2aea53ecbf6b6513e9d9887013660064afa5d2caa6b37e01d32f6d40d63f: Status 404 returned error can't find the container with id 6b0f2aea53ecbf6b6513e9d9887013660064afa5d2caa6b37e01d32f6d40d63f Sep 17 02:08:13.660885 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.660859 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" event={"ID":"0b9a5e67-629f-49a3-aaaf-0dea8170d80f","Type":"ContainerStarted","Data":"6b0f2aea53ecbf6b6513e9d9887013660064afa5d2caa6b37e01d32f6d40d63f"} Sep 17 02:08:13.975189 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:13.975164 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:08:14.060937 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:14.060905 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:14.667380 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:14.667296 2582 generic.go:356] "Generic (PLEG): container finished" podID="0b9a5e67-629f-49a3-aaaf-0dea8170d80f" containerID="ab250d18abde5209a1e157516ce5685530055856057b2d800035c1af79326ae0" exitCode=0 Sep 17 02:08:14.667719 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:14.667390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" event={"ID":"0b9a5e67-629f-49a3-aaaf-0dea8170d80f","Type":"ContainerDied","Data":"ab250d18abde5209a1e157516ce5685530055856057b2d800035c1af79326ae0"} Sep 17 02:08:14.762438 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:14.762402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:15.178726 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.178699 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb"] Sep 17 02:08:15.181736 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.181712 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.184819 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.184792 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"default-dockercfg-xb9xt\"" Sep 17 02:08:15.184952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.184854 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"rate-test-model-f03337-kserve-self-signed-certs\"" Sep 17 02:08:15.184952 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.184870 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"openshift-service-ca.crt\"" Sep 17 02:08:15.185069 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.184980 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f03337\"/\"kube-root-ca.crt\"" Sep 17 02:08:15.191023 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.191001 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb"] Sep 17 02:08:15.200298 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.200268 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:15.289959 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.289939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-dshm\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.290061 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.289971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738c127-6eac-4307-b910-225859534d54-tls-certs\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.290061 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.290011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-model-cache\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.290193 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.290101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2pf\" (UniqueName: \"kubernetes.io/projected/e738c127-6eac-4307-b910-225859534d54-kube-api-access-xv2pf\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.290193 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.290148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-home\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.290320 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.290243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-tmp-dir\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391072 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-tmp-dir\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391201 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-dshm\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391201 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391103 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738c127-6eac-4307-b910-225859534d54-tls-certs\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391201 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-model-cache\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391201 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2pf\" (UniqueName: \"kubernetes.io/projected/e738c127-6eac-4307-b910-225859534d54-kube-api-access-xv2pf\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391201 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-home\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391594 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391566 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-model-cache\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391667 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-tmp-dir\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.391667 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.391632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-home\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.393382 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.393364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-dshm\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.393686 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.393668 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738c127-6eac-4307-b910-225859534d54-tls-certs\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.398911 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.398889 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2pf\" (UniqueName: \"kubernetes.io/projected/e738c127-6eac-4307-b910-225859534d54-kube-api-access-xv2pf\") pod \"rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.491677 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.491656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:15.554842 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.554814 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:15.763546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.763523 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb"] Sep 17 02:08:15.765423 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:08:15.765398 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode738c127_6eac_4307_b910_225859534d54.slice/crio-22f455239f1522e85a2af0857e44f7aecdcba7cd00e7120505217ca23aae866f WatchSource:0}: Error finding container 22f455239f1522e85a2af0857e44f7aecdcba7cd00e7120505217ca23aae866f: Status 404 returned error can't find the container with id 22f455239f1522e85a2af0857e44f7aecdcba7cd00e7120505217ca23aae866f Sep 17 02:08:15.811958 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.811937 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:15.895583 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.895544 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a5e67-629f-49a3-aaaf-0dea8170d80f-maas-api-service-ca\") pod \"0b9a5e67-629f-49a3-aaaf-0dea8170d80f\" (UID: \"0b9a5e67-629f-49a3-aaaf-0dea8170d80f\") " Sep 17 02:08:15.895873 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.895850 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9a5e67-629f-49a3-aaaf-0dea8170d80f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b9a5e67-629f-49a3-aaaf-0dea8170d80f" (UID: "0b9a5e67-629f-49a3-aaaf-0dea8170d80f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:15.895989 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:15.895971 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a5e67-629f-49a3-aaaf-0dea8170d80f-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:16.543007 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.542974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:16.680109 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.680024 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" event={"ID":"e738c127-6eac-4307-b910-225859534d54","Type":"ContainerStarted","Data":"50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a"} Sep 17 02:08:16.680109 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.680069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" event={"ID":"e738c127-6eac-4307-b910-225859534d54","Type":"ContainerStarted","Data":"22f455239f1522e85a2af0857e44f7aecdcba7cd00e7120505217ca23aae866f"} Sep 17 02:08:16.681332 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.681308 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" event={"ID":"0b9a5e67-629f-49a3-aaaf-0dea8170d80f","Type":"ContainerDied","Data":"6b0f2aea53ecbf6b6513e9d9887013660064afa5d2caa6b37e01d32f6d40d63f"} Sep 17 02:08:16.681443 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.681347 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0f2aea53ecbf6b6513e9d9887013660064afa5d2caa6b37e01d32f6d40d63f" Sep 17 02:08:16.681443 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.681315 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9" Sep 17 02:08:16.694360 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.694323 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" podStartSLOduration=1.6943104660000001 podStartE2EDuration="1.694310466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:08:16.693996312 +0000 UTC m=+1841.529048647" watchObservedRunningTime="2026-09-17 02:08:16.694310466 +0000 UTC m=+1841.529362799" Sep 17 02:08:16.870948 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.870912 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 02:08:16.871292 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:16.870975 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 02:08:17.688905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.688872 2582 generic.go:356] "Generic (PLEG): container finished" podID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerID="e0a13f6917a514c597635545c4f1ef59a6bacd34005207718097eec4ca86586b" exitCode=0 Sep 17 02:08:17.689014 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.688938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" event={"ID":"8a9fb5af-35ba-4549-9a10-62cdc030046a","Type":"ContainerDied","Data":"e0a13f6917a514c597635545c4f1ef59a6bacd34005207718097eec4ca86586b"} Sep 17 02:08:17.689014 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.688997 2582 scope.go:117] "RemoveContainer" containerID="aa13f9b143f50668544d623c5e532c4b6ffb17e79576df08579b880a849ae02d" Sep 17 02:08:17.813846 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.813816 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:08:17.914730 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914667 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-certs\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.914730 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914702 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-socket\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.914730 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914725 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm4kp\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-kube-api-access-rm4kp\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914764 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-token\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914794 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-podinfo\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914839 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-envoy\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914869 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-data\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.914982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-credential-socket\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915076 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9fb5af-35ba-4549-9a10-62cdc030046a-istiod-ca-cert\") pod \"8a9fb5af-35ba-4549-9a10-62cdc030046a\" (UID: \"8a9fb5af-35ba-4549-9a10-62cdc030046a\") " Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915270 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915288 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915483 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-data\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915483 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915502 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915517 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-workload-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:17.915651 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.915586 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9fb5af-35ba-4549-9a10-62cdc030046a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:17.917587 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.917562 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-kube-api-access-rm4kp" (OuterVolumeSpecName: "kube-api-access-rm4kp") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "kube-api-access-rm4kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:17.917709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.917598 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:17.917786 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.917770 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:17.917846 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:17.917810 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a9fb5af-35ba-4549-9a10-62cdc030046a" (UID: "8a9fb5af-35ba-4549-9a10-62cdc030046a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 02:08:18.016636 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.016610 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-credential-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:18.016735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.016639 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9fb5af-35ba-4549-9a10-62cdc030046a-istiod-ca-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:18.016735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.016654 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4kp\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-kube-api-access-rm4kp\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:18.016735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.016667 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:18.016735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.016681 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-podinfo\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:18.016735 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.016694 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9fb5af-35ba-4549-9a10-62cdc030046a-istio-envoy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:18.094686 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.094660 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:18.698044 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.698012 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" event={"ID":"8a9fb5af-35ba-4549-9a10-62cdc030046a","Type":"ContainerDied","Data":"b27129862cfc473dcb9c41d2d8cf7f5d3a6d1322ab72d8d686661de6aba67a87"} Sep 17 02:08:18.698196 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.698061 2582 scope.go:117] "RemoveContainer" containerID="e0a13f6917a514c597635545c4f1ef59a6bacd34005207718097eec4ca86586b" Sep 17 02:08:18.698196 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.698019 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth" Sep 17 02:08:18.718997 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.718970 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth"] Sep 17 02:08:18.723098 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:18.723077 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c5da3b-openshift-default-764cdd779b-gkjth"] Sep 17 02:08:19.679432 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:19.679400 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" path="/var/lib/kubelet/pods/8a9fb5af-35ba-4549-9a10-62cdc030046a/volumes" Sep 17 02:08:23.566091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.566051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:23.587167 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.587140 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5"] Sep 17 02:08:23.587474 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.587450 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" podUID="7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" containerName="payload-processing" containerID="cri-o://1101018e6f6275c8928db76a0e92522d74c4bf33e3f990424a68fb6ead9ebfe8" gracePeriod=30 Sep 17 02:08:23.597159 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.597132 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f"] Sep 17 02:08:23.597399 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.597375 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" podUID="9c9a3fcc-90ee-4880-93e9-b58fd61c882b" containerName="payload-pre-processing" containerID="cri-o://6d4ac353527f3c736ed5fce7307262f36f7932cdf44ef59c340ab8127b82f874" gracePeriod=30 Sep 17 02:08:23.632539 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.632511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:23.695652 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.695620 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:23.727031 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.726999 2582 generic.go:356] "Generic (PLEG): container finished" podID="7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" containerID="1101018e6f6275c8928db76a0e92522d74c4bf33e3f990424a68fb6ead9ebfe8" exitCode=0 Sep 17 02:08:23.727274 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.727248 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" event={"ID":"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54","Type":"ContainerDied","Data":"1101018e6f6275c8928db76a0e92522d74c4bf33e3f990424a68fb6ead9ebfe8"} Sep 17 02:08:23.728794 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.728771 2582 generic.go:356] "Generic (PLEG): container finished" podID="9c9a3fcc-90ee-4880-93e9-b58fd61c882b" containerID="6d4ac353527f3c736ed5fce7307262f36f7932cdf44ef59c340ab8127b82f874" exitCode=0 Sep 17 02:08:23.728894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.728807 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" event={"ID":"9c9a3fcc-90ee-4880-93e9-b58fd61c882b","Type":"ContainerDied","Data":"6d4ac353527f3c736ed5fce7307262f36f7932cdf44ef59c340ab8127b82f874"} Sep 17 02:08:23.754408 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.754367 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9"] Sep 17 02:08:23.756405 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.756344 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a3912d-hxrd9"] Sep 17 02:08:23.803741 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.803696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:23.880587 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.880559 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:08:23.882630 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.882610 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:08:23.950512 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.950475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:23.965749 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.965712 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgf8x\" (UniqueName: \"kubernetes.io/projected/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-kube-api-access-dgf8x\") pod \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " Sep 17 02:08:23.965937 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.965788 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-plugins-config-volume\") pod \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\" (UID: \"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54\") " Sep 17 02:08:23.966303 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.966274 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" (UID: "7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:23.967931 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:23.967902 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-kube-api-access-dgf8x" (OuterVolumeSpecName: "kube-api-access-dgf8x") pod "7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" (UID: "7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54"). InnerVolumeSpecName "kube-api-access-dgf8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:24.066929 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.066903 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-plugins-config-volume\") pod \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " Sep 17 02:08:24.067039 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.066962 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr2fq\" (UniqueName: \"kubernetes.io/projected/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-kube-api-access-kr2fq\") pod \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\" (UID: \"9c9a3fcc-90ee-4880-93e9-b58fd61c882b\") " Sep 17 02:08:24.067187 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.067175 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgf8x\" (UniqueName: \"kubernetes.io/projected/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-kube-api-access-dgf8x\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:24.067248 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.067189 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:24.067306 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.067285 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c9a3fcc-90ee-4880-93e9-b58fd61c882b" (UID: "9c9a3fcc-90ee-4880-93e9-b58fd61c882b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:24.068929 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.068902 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-kube-api-access-kr2fq" (OuterVolumeSpecName: "kube-api-access-kr2fq") pod "9c9a3fcc-90ee-4880-93e9-b58fd61c882b" (UID: "9c9a3fcc-90ee-4880-93e9-b58fd61c882b"). InnerVolumeSpecName "kube-api-access-kr2fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:24.150824 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.150759 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:24.167972 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.167950 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2fq\" (UniqueName: \"kubernetes.io/projected/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-kube-api-access-kr2fq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:24.167972 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.167968 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a3fcc-90ee-4880-93e9-b58fd61c882b-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:24.219293 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.219262 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:24.714896 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.714809 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:24.735775 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.735749 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" Sep 17 02:08:24.735911 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.735797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5" event={"ID":"7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54","Type":"ContainerDied","Data":"9b641f002fd3db3071ea403d53fcaa7a19c4e9202e86f6a4372887f495258c25"} Sep 17 02:08:24.735911 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.735850 2582 scope.go:117] "RemoveContainer" containerID="1101018e6f6275c8928db76a0e92522d74c4bf33e3f990424a68fb6ead9ebfe8" Sep 17 02:08:24.737329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.737308 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" event={"ID":"9c9a3fcc-90ee-4880-93e9-b58fd61c882b","Type":"ContainerDied","Data":"4db8ab3b61387d0c222aa92664b8a19121670c464cdd738cac62859c8168fb87"} Sep 17 02:08:24.737407 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.737338 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f" Sep 17 02:08:24.746545 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.746528 2582 scope.go:117] "RemoveContainer" containerID="6d4ac353527f3c736ed5fce7307262f36f7932cdf44ef59c340ab8127b82f874" Sep 17 02:08:24.759332 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.759306 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5"] Sep 17 02:08:24.761288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.761271 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a3912d-66dc8d6df-gcxd5"] Sep 17 02:08:24.769837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.769817 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f"] Sep 17 02:08:24.772722 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.772700 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:08:24.776093 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:24.776072 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a3912d-9b7d69b66-ndw4f"] Sep 17 02:08:25.492592 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.492553 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:25.492592 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.492602 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:25.505346 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.505317 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:25.620772 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.620745 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:25.680131 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.680101 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b9a5e67-629f-49a3-aaaf-0dea8170d80f" path="/var/lib/kubelet/pods/0b9a5e67-629f-49a3-aaaf-0dea8170d80f/volumes" Sep 17 02:08:25.680491 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.680477 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" path="/var/lib/kubelet/pods/7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54/volumes" Sep 17 02:08:25.680800 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.680788 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9a3fcc-90ee-4880-93e9-b58fd61c882b" path="/var/lib/kubelet/pods/9c9a3fcc-90ee-4880-93e9-b58fd61c882b/volumes" Sep 17 02:08:25.756258 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:25.756175 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:08:30.033641 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:30.033607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:30.134540 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:30.134500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:30.275420 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:30.275382 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5"] Sep 17 02:08:30.275739 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:30.275706 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" containerID="cri-o://292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1" gracePeriod=30 Sep 17 02:08:30.286596 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:30.286518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:35.537239 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.537152 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:08:35.567204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567177 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-data\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567238 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-certs\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567256 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-credential-socket\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567301 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-token\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567333 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-788gk\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-kube-api-access-788gk\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567378 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-podinfo\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567424 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istiod-ca-cert\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567447 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:35.567662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567460 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-socket\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567540 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:35.567662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567556 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:35.567662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567551 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-envoy\") pod \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\" (UID: \"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478\") " Sep 17 02:08:35.567928 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567845 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:08:35.567984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567931 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istiod-ca-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.567984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567953 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-data\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.567984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567966 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.567984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.567979 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-credential-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.568185 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.568125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:35.569656 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.569623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:35.569773 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.569657 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 02:08:35.570013 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.569995 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:08:35.570052 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.570028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-kube-api-access-788gk" (OuterVolumeSpecName: "kube-api-access-788gk") pod "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" (UID: "7c0bff41-99a0-46c3-9b4f-7b17f1ce2478"). InnerVolumeSpecName "kube-api-access-788gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:08:35.668633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.668599 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-workload-socket\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.668633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.668628 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-envoy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.668633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.668639 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.668633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.668647 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788gk\" (UniqueName: \"kubernetes.io/projected/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-kube-api-access-788gk\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.668914 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.668656 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478-istio-podinfo\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:08:35.797827 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.797726 2582 generic.go:356] "Generic (PLEG): container finished" podID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerID="292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1" exitCode=0 Sep 17 02:08:35.797827 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.797796 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" Sep 17 02:08:35.797827 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.797806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" event={"ID":"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478","Type":"ContainerDied","Data":"292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1"} Sep 17 02:08:35.798089 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.797871 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5" event={"ID":"7c0bff41-99a0-46c3-9b4f-7b17f1ce2478","Type":"ContainerDied","Data":"1f8b9be145c935cad5b86bc02c6be330bda710fa066101f23ceac1dd62959601"} Sep 17 02:08:35.798089 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.797886 2582 scope.go:117] "RemoveContainer" containerID="292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1" Sep 17 02:08:35.807901 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.807877 2582 scope.go:117] "RemoveContainer" containerID="cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d" Sep 17 02:08:35.814965 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.814920 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5"] Sep 17 02:08:35.818506 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.818472 2582 scope.go:117] "RemoveContainer" containerID="292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1" Sep 17 02:08:35.819000 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:08:35.818968 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1\": container with ID starting with 292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1 not found: ID does not exist" containerID="292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1" Sep 17 02:08:35.819115 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.819011 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1"} err="failed to get container status \"292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1\": rpc error: code = NotFound desc = could not find container \"292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1\": container with ID starting with 292fc00e14173aafbf86b7aa209f32ed29603ce30df21604259bda857ad2e3c1 not found: ID does not exist" Sep 17 02:08:35.819115 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.819039 2582 scope.go:117] "RemoveContainer" containerID="cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d" Sep 17 02:08:35.819402 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:08:35.819379 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d\": container with ID starting with cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d not found: ID does not exist" containerID="cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d" Sep 17 02:08:35.819489 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.819407 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d"} err="failed to get container status \"cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d\": rpc error: code = NotFound desc = could not find container \"cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d\": container with ID starting with cbb8c7f815a5235fdbc400fdf06be09a4fb0a132ff6154364355ba3a0a66454d not found: ID does not exist" Sep 17 02:08:35.820592 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:35.820565 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a3912d-openshift-default-5c47b47c8-g4vz5"] Sep 17 02:08:37.680093 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:37.680054 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" path="/var/lib/kubelet/pods/7c0bff41-99a0-46c3-9b4f-7b17f1ce2478/volumes" Sep 17 02:08:38.965596 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:38.965564 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:08:47.968746 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:47.968712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:08:54.701017 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:54.700934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:55.560449 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:55.560415 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb"] Sep 17 02:08:55.576954 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:55.576917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:56.109672 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:56.109631 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:56.374272 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:56.374168 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:56.577220 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:56.577183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:58.109304 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:58.109221 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:08:58.419819 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:58.419721 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:59.105023 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:59.104991 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:59.253685 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:59.253646 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:59.349637 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:59.349597 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:08:59.935590 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:08:59.935539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:00.862993 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:00.862956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:00.922150 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:00.922119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:01.046350 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.046311 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:01.343324 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.343289 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:01.417535 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.417504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:01.455667 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.455633 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7"] Sep 17 02:09:01.456037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456019 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b9a5e67-629f-49a3-aaaf-0dea8170d80f" containerName="revoke-keys" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456039 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b9a5e67-629f-49a3-aaaf-0dea8170d80f" containerName="revoke-keys" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456063 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456072 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456083 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456093 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456107 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" containerName="payload-processing" Sep 17 02:09:01.456123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456115 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" containerName="payload-processing" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456137 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9a3fcc-90ee-4880-93e9-b58fd61c882b" containerName="payload-pre-processing" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456145 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9a3fcc-90ee-4880-93e9-b58fd61c882b" containerName="payload-pre-processing" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456249 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9a3fcc-90ee-4880-93e9-b58fd61c882b" containerName="payload-pre-processing" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456265 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456277 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456287 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456298 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9a6bf2-1df9-4372-9f9b-d26d5a30eb54" containerName="payload-processing" Sep 17 02:09:01.456515 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.456309 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b9a5e67-629f-49a3-aaaf-0dea8170d80f" containerName="revoke-keys" Sep 17 02:09:01.459656 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.459636 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.461916 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.461897 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0d7c34-dockercfg-drjlq\"" Sep 17 02:09:01.461916 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.461910 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0d7c34\"" Sep 17 02:09:01.467139 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.466959 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7"] Sep 17 02:09:01.473002 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.472983 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb"] Sep 17 02:09:01.473320 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.473308 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" Sep 17 02:09:01.473370 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.473321 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" Sep 17 02:09:01.473370 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.473330 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" Sep 17 02:09:01.473370 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.473336 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c0bff41-99a0-46c3-9b4f-7b17f1ce2478" containerName="istio-proxy" Sep 17 02:09:01.473467 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.473381 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9fb5af-35ba-4549-9a10-62cdc030046a" containerName="istio-proxy" Sep 17 02:09:01.476131 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.476117 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.485879 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.485859 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb"] Sep 17 02:09:01.603971 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.603901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91374ce7-81b0-4db5-8bb4-49e10cd367de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.603971 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.603952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfc0e30-9a18-4796-a358-e60688b6bbef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.604121 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.604004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7qn\" (UniqueName: \"kubernetes.io/projected/91374ce7-81b0-4db5-8bb4-49e10cd367de-kube-api-access-vc7qn\") pod \"payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.604121 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.604082 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6r67\" (UniqueName: \"kubernetes.io/projected/6dfc0e30-9a18-4796-a358-e60688b6bbef-kube-api-access-t6r67\") pod \"payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.705157 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.705132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r67\" (UniqueName: \"kubernetes.io/projected/6dfc0e30-9a18-4796-a358-e60688b6bbef-kube-api-access-t6r67\") pod \"payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.705305 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.705175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91374ce7-81b0-4db5-8bb4-49e10cd367de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.705305 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.705247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfc0e30-9a18-4796-a358-e60688b6bbef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.705305 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.705269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7qn\" (UniqueName: \"kubernetes.io/projected/91374ce7-81b0-4db5-8bb4-49e10cd367de-kube-api-access-vc7qn\") pod \"payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.705887 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.705869 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91374ce7-81b0-4db5-8bb4-49e10cd367de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.705966 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.705948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfc0e30-9a18-4796-a358-e60688b6bbef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.712575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.712546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r67\" (UniqueName: \"kubernetes.io/projected/6dfc0e30-9a18-4796-a358-e60688b6bbef-kube-api-access-t6r67\") pod \"payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:01.712743 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.712721 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7qn\" (UniqueName: \"kubernetes.io/projected/91374ce7-81b0-4db5-8bb4-49e10cd367de-kube-api-access-vc7qn\") pod \"payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.770998 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.770975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:01.788794 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:01.788771 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:02.044660 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.044637 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7"] Sep 17 02:09:02.046601 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:09:02.046577 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91374ce7_81b0_4db5_8bb4_49e10cd367de.slice/crio-50a8ae88366ca0662fb38972d2f1da993cb88a99b98708223c32d64a7d8ddf42 WatchSource:0}: Error finding container 50a8ae88366ca0662fb38972d2f1da993cb88a99b98708223c32d64a7d8ddf42: Status 404 returned error can't find the container with id 50a8ae88366ca0662fb38972d2f1da993cb88a99b98708223c32d64a7d8ddf42 Sep 17 02:09:02.067257 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.067234 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb"] Sep 17 02:09:02.068772 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:09:02.068741 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfc0e30_9a18_4796_a358_e60688b6bbef.slice/crio-8aa1bf157da4c05083af0e8b113d4445acaed56cea303a50b5d826782635e51e WatchSource:0}: Error finding container 8aa1bf157da4c05083af0e8b113d4445acaed56cea303a50b5d826782635e51e: Status 404 returned error can't find the container with id 8aa1bf157da4c05083af0e8b113d4445acaed56cea303a50b5d826782635e51e Sep 17 02:09:02.937176 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.937145 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" event={"ID":"91374ce7-81b0-4db5-8bb4-49e10cd367de","Type":"ContainerStarted","Data":"534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74"} Sep 17 02:09:02.937350 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.937194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" event={"ID":"91374ce7-81b0-4db5-8bb4-49e10cd367de","Type":"ContainerStarted","Data":"50a8ae88366ca0662fb38972d2f1da993cb88a99b98708223c32d64a7d8ddf42"} Sep 17 02:09:02.937350 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.937252 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:02.938722 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.938700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" event={"ID":"6dfc0e30-9a18-4796-a358-e60688b6bbef","Type":"ContainerStarted","Data":"596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146"} Sep 17 02:09:02.938834 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.938742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" event={"ID":"6dfc0e30-9a18-4796-a358-e60688b6bbef","Type":"ContainerStarted","Data":"8aa1bf157da4c05083af0e8b113d4445acaed56cea303a50b5d826782635e51e"} Sep 17 02:09:02.938834 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.938770 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:02.954585 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.954534 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" podStartSLOduration=1.9545153499999999 podStartE2EDuration="1.95451535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:09:02.952383709 +0000 UTC m=+1887.787436044" watchObservedRunningTime="2026-09-17 02:09:02.95451535 +0000 UTC m=+1887.789567684" Sep 17 02:09:02.970288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:02.970186 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" podStartSLOduration=1.970166063 podStartE2EDuration="1.970166063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:09:02.966726203 +0000 UTC m=+1887.801778530" watchObservedRunningTime="2026-09-17 02:09:02.970166063 +0000 UTC m=+1887.805218400" Sep 17 02:09:10.579317 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.579277 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" podUID="e738c127-6eac-4307-b910-225859534d54" containerName="main" containerID="cri-o://50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a" gracePeriod=45 Sep 17 02:09:10.831221 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.831148 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:09:10.975525 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975499 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-tmp-dir\") pod \"e738c127-6eac-4307-b910-225859534d54\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " Sep 17 02:09:10.975675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975534 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738c127-6eac-4307-b910-225859534d54-tls-certs\") pod \"e738c127-6eac-4307-b910-225859534d54\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " Sep 17 02:09:10.975675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975560 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-model-cache\") pod \"e738c127-6eac-4307-b910-225859534d54\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " Sep 17 02:09:10.975675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975585 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv2pf\" (UniqueName: \"kubernetes.io/projected/e738c127-6eac-4307-b910-225859534d54-kube-api-access-xv2pf\") pod \"e738c127-6eac-4307-b910-225859534d54\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " Sep 17 02:09:10.975675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975613 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-dshm\") pod \"e738c127-6eac-4307-b910-225859534d54\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " Sep 17 02:09:10.975912 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975699 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-home\") pod \"e738c127-6eac-4307-b910-225859534d54\" (UID: \"e738c127-6eac-4307-b910-225859534d54\") " Sep 17 02:09:10.975912 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975806 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-model-cache" (OuterVolumeSpecName: "model-cache") pod "e738c127-6eac-4307-b910-225859534d54" (UID: "e738c127-6eac-4307-b910-225859534d54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:09:10.975912 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.975819 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e738c127-6eac-4307-b910-225859534d54" (UID: "e738c127-6eac-4307-b910-225859534d54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:09:10.976075 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.976004 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-home" (OuterVolumeSpecName: "home") pod "e738c127-6eac-4307-b910-225859534d54" (UID: "e738c127-6eac-4307-b910-225859534d54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:09:10.976075 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.976044 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:09:10.976075 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.976059 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:09:10.977808 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.977781 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-dshm" (OuterVolumeSpecName: "dshm") pod "e738c127-6eac-4307-b910-225859534d54" (UID: "e738c127-6eac-4307-b910-225859534d54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:09:10.977913 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.977808 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e738c127-6eac-4307-b910-225859534d54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e738c127-6eac-4307-b910-225859534d54" (UID: "e738c127-6eac-4307-b910-225859534d54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:09:10.978053 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.978034 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e738c127-6eac-4307-b910-225859534d54-kube-api-access-xv2pf" (OuterVolumeSpecName: "kube-api-access-xv2pf") pod "e738c127-6eac-4307-b910-225859534d54" (UID: "e738c127-6eac-4307-b910-225859534d54"). InnerVolumeSpecName "kube-api-access-xv2pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:09:10.979979 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.979953 2582 generic.go:356] "Generic (PLEG): container finished" podID="e738c127-6eac-4307-b910-225859534d54" containerID="50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a" exitCode=0 Sep 17 02:09:10.980102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.979986 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" event={"ID":"e738c127-6eac-4307-b910-225859534d54","Type":"ContainerDied","Data":"50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a"} Sep 17 02:09:10.980102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.979997 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" Sep 17 02:09:10.980102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.980013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb" event={"ID":"e738c127-6eac-4307-b910-225859534d54","Type":"ContainerDied","Data":"22f455239f1522e85a2af0857e44f7aecdcba7cd00e7120505217ca23aae866f"} Sep 17 02:09:10.980102 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.980029 2582 scope.go:117] "RemoveContainer" containerID="50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a" Sep 17 02:09:10.996204 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.996187 2582 scope.go:117] "RemoveContainer" containerID="50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a" Sep 17 02:09:10.996499 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:09:10.996479 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a\": container with ID starting with 50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a not found: ID does not exist" containerID="50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a" Sep 17 02:09:10.996569 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:10.996509 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a"} err="failed to get container status \"50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a\": rpc error: code = NotFound desc = could not find container \"50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a\": container with ID starting with 50c375a16dfca1e04fd5b0d3fc42a2de4a178fccdc1b7179956eb89ee301243a not found: ID does not exist" Sep 17 02:09:11.007634 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.007611 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb"] Sep 17 02:09:11.012642 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.012622 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f03337/rate-test-model-f03337-kserve-6dcc44d6c5-dhsrb"] Sep 17 02:09:11.077283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.077257 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:09:11.077283 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.077280 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738c127-6eac-4307-b910-225859534d54-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:09:11.077418 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.077295 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv2pf\" (UniqueName: \"kubernetes.io/projected/e738c127-6eac-4307-b910-225859534d54-kube-api-access-xv2pf\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:09:11.077418 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.077308 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738c127-6eac-4307-b910-225859534d54-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:09:11.684486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.684457 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e738c127-6eac-4307-b910-225859534d54" path="/var/lib/kubelet/pods/e738c127-6eac-4307-b910-225859534d54/volumes" Sep 17 02:09:11.776545 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:11.776509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:09:13.946372 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:13.946337 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:09:13.946837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:13.946410 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:09:22.852728 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:22.852696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:23.647491 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:23.647452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:24.195575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:24.195536 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:24.320336 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:24.320288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:24.938770 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:24.938733 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:25.041242 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:25.041192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:25.263657 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:25.263625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:25.621953 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:25.621870 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:27.628508 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:27.628466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:28.297595 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:28.297566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:28.388946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:28.388917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:29.047454 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:29.047422 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:29.795562 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:29.795522 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:30.555408 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:30.555376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:30.611920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:30.611884 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:30.720221 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:30.720177 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:31.011366 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.011330 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:31.106668 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.106637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:31.120522 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.120496 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc"] Sep 17 02:09:31.121078 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.121058 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e738c127-6eac-4307-b910-225859534d54" containerName="main" Sep 17 02:09:31.121131 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.121083 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e738c127-6eac-4307-b910-225859534d54" containerName="main" Sep 17 02:09:31.121198 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.121187 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e738c127-6eac-4307-b910-225859534d54" containerName="main" Sep 17 02:09:31.125974 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.125957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.129310 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.129255 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4ee767-dockercfg-8qwv9\"" Sep 17 02:09:31.129770 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.129753 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4ee767\"" Sep 17 02:09:31.131495 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.131475 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc"] Sep 17 02:09:31.233726 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.233702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a4e869-0117-437e-9815-9738bd2edc77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.233816 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.233746 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/31a4e869-0117-437e-9815-9738bd2edc77-kube-api-access-mz56b\") pod \"payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.335033 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.334961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a4e869-0117-437e-9815-9738bd2edc77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.335033 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.335008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/31a4e869-0117-437e-9815-9738bd2edc77-kube-api-access-mz56b\") pod \"payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.335662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.335642 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a4e869-0117-437e-9815-9738bd2edc77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.342602 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.342583 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/31a4e869-0117-437e-9815-9738bd2edc77-kube-api-access-mz56b\") pod \"payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.438055 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.438030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:31.708647 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:31.708622 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc"] Sep 17 02:09:31.710970 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:09:31.710940 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a4e869_0117_437e_9815_9738bd2edc77.slice/crio-8c813eda7435b7fa9eea2fd783493cd012f52a8ee7ffb2ea86a8a0632f48a1a2 WatchSource:0}: Error finding container 8c813eda7435b7fa9eea2fd783493cd012f52a8ee7ffb2ea86a8a0632f48a1a2: Status 404 returned error can't find the container with id 8c813eda7435b7fa9eea2fd783493cd012f52a8ee7ffb2ea86a8a0632f48a1a2 Sep 17 02:09:32.092198 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:32.092163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" event={"ID":"31a4e869-0117-437e-9815-9738bd2edc77","Type":"ContainerStarted","Data":"05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2"} Sep 17 02:09:32.092408 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:32.092242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" event={"ID":"31a4e869-0117-437e-9815-9738bd2edc77","Type":"ContainerStarted","Data":"8c813eda7435b7fa9eea2fd783493cd012f52a8ee7ffb2ea86a8a0632f48a1a2"} Sep 17 02:09:32.092408 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:32.092272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:32.108396 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:32.108347 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" podStartSLOduration=1.108332525 podStartE2EDuration="1.108332525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:09:32.106578898 +0000 UTC m=+1916.941631273" watchObservedRunningTime="2026-09-17 02:09:32.108332525 +0000 UTC m=+1916.943384858" Sep 17 02:09:36.170253 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:36.170219 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:09:43.099156 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:43.099129 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:09:45.090108 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:45.090071 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:52.638675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:52.638630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:53.391833 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:53.391797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:53.492111 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:53.492075 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:54.284829 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:54.284791 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:55.101908 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:55.101876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:09:55.641932 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:09:55.641903 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:11.177863 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.177825 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms"] Sep 17 02:10:11.181096 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.181073 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.184617 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.184598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0d7c34\"/\"default-dockercfg-h95lr\"" Sep 17 02:10:11.184837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.184600 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0d7c34\"/\"kube-root-ca.crt\"" Sep 17 02:10:11.184993 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.184649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0d7c34\"/\"ipp-route-0d7c34-kserve-self-signed-certs\"" Sep 17 02:10:11.185109 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.184649 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0d7c34\"/\"openshift-service-ca.crt\"" Sep 17 02:10:11.191980 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.191952 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms"] Sep 17 02:10:11.204434 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.204406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:11.248183 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.248160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tmp-dir\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.248307 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.248203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/0905a014-0f8c-4171-a0e2-2c5c426b11b3-kube-api-access-nzw8d\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.248307 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.248268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tls-certs\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.248434 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.248309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-model-cache\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.248434 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.248367 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-dshm\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.248434 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.248416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-home\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.349726 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.349698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-home\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.349840 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.349758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tmp-dir\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.349840 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.349808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/0905a014-0f8c-4171-a0e2-2c5c426b11b3-kube-api-access-nzw8d\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.349840 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.349836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tls-certs\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.350009 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.349865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-model-cache\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.350009 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.349941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-dshm\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.350116 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.350076 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-home\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.350179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.350127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tmp-dir\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.350261 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.350185 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-model-cache\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.352020 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.352002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-dshm\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.352329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.352311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tls-certs\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.356803 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.356783 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/0905a014-0f8c-4171-a0e2-2c5c426b11b3-kube-api-access-nzw8d\") pod \"ipp-route-0d7c34-kserve-f998bdfcf-zhvms\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.491662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.491644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:11.765168 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:11.761723 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms"] Sep 17 02:10:11.767546 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:10:11.767516 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0905a014_0f8c_4171_a0e2_2c5c426b11b3.slice/crio-ff71d184442b636fe84886a3d1691bff1f131ff744fb112a4058344c9c866caf WatchSource:0}: Error finding container ff71d184442b636fe84886a3d1691bff1f131ff744fb112a4058344c9c866caf: Status 404 returned error can't find the container with id ff71d184442b636fe84886a3d1691bff1f131ff744fb112a4058344c9c866caf Sep 17 02:10:12.297311 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:12.297275 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" event={"ID":"0905a014-0f8c-4171-a0e2-2c5c426b11b3","Type":"ContainerStarted","Data":"c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1"} Sep 17 02:10:12.297718 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:12.297324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" event={"ID":"0905a014-0f8c-4171-a0e2-2c5c426b11b3","Type":"ContainerStarted","Data":"ff71d184442b636fe84886a3d1691bff1f131ff744fb112a4058344c9c866caf"} Sep 17 02:10:12.315353 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:12.315301 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" podStartSLOduration=1.315283693 podStartE2EDuration="1.315283693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:10:12.312751583 +0000 UTC m=+1957.147803918" watchObservedRunningTime="2026-09-17 02:10:12.315283693 +0000 UTC m=+1957.150336026" Sep 17 02:10:15.073710 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:15.073672 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:10:21.492187 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:21.492150 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:21.492187 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:21.492194 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:21.504585 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:21.504564 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:22.364933 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:22.364904 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:10:29.278669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:29.278577 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:10:29.619663 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:29.619563 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:30.841335 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:30.841303 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:35.533021 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:35.532981 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb_6dfc0e30-9a18-4796-a358-e60688b6bbef/payload-processing/0.log" Sep 17 02:10:35.997831 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:35.997803 2582 scope.go:117] "RemoveContainer" containerID="1489734c970f9ded7bee52e31daa460a1c8f5f21d8e49673c29fda1f8b196785" Sep 17 02:10:36.006117 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:36.006092 2582 scope.go:117] "RemoveContainer" containerID="9ab8423b37533f76b341ceca664113a0ce33bb6776b4661d59338cf03c43207e" Sep 17 02:10:36.332274 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:36.332174 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:36.689170 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:36.689075 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:36.960200 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:36.960120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:39.343123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:39.343078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:39.671040 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:39.670950 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:39.775172 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:39.775128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:40.239451 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:40.239416 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:41.106781 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:41.106743 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:41.930196 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:41.930146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:42.642556 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:42.642521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:42.743901 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:42.743854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:42.877267 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:42.877203 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:43.101619 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.101581 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:43.204927 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.204891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:43.211658 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.211631 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv"] Sep 17 02:10:43.215057 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.215041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.217471 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.217451 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-557e7f-dockercfg-9kn27\"" Sep 17 02:10:43.217642 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.217620 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-557e7f\"" Sep 17 02:10:43.225560 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.225537 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv"] Sep 17 02:10:43.293135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.293111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7d6f4-7b26-49b1-8856-07055f476554-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.293275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.293145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2v6b\" (UniqueName: \"kubernetes.io/projected/7bd7d6f4-7b26-49b1-8856-07055f476554-kube-api-access-v2v6b\") pod \"payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.394712 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.394635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7d6f4-7b26-49b1-8856-07055f476554-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.398223 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.394891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2v6b\" (UniqueName: \"kubernetes.io/projected/7bd7d6f4-7b26-49b1-8856-07055f476554-kube-api-access-v2v6b\") pod \"payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.400494 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.400426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7d6f4-7b26-49b1-8856-07055f476554-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.403335 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.403314 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2v6b\" (UniqueName: \"kubernetes.io/projected/7bd7d6f4-7b26-49b1-8856-07055f476554-kube-api-access-v2v6b\") pod \"payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.526527 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.526504 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:43.796843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:43.796815 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv"] Sep 17 02:10:43.799181 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:10:43.799149 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd7d6f4_7b26_49b1_8856_07055f476554.slice/crio-4eea115387512f818b28e358a257e6c0aa9da922cbcc5ef2211dd9163e6f771f WatchSource:0}: Error finding container 4eea115387512f818b28e358a257e6c0aa9da922cbcc5ef2211dd9163e6f771f: Status 404 returned error can't find the container with id 4eea115387512f818b28e358a257e6c0aa9da922cbcc5ef2211dd9163e6f771f Sep 17 02:10:44.469546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:44.469512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" event={"ID":"7bd7d6f4-7b26-49b1-8856-07055f476554","Type":"ContainerStarted","Data":"386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064"} Sep 17 02:10:44.469825 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:44.469566 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" event={"ID":"7bd7d6f4-7b26-49b1-8856-07055f476554","Type":"ContainerStarted","Data":"4eea115387512f818b28e358a257e6c0aa9da922cbcc5ef2211dd9163e6f771f"} Sep 17 02:10:44.469825 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:44.469599 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:10:44.484755 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:44.484710 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" podStartSLOduration=1.484696774 podStartE2EDuration="1.484696774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:10:44.48408699 +0000 UTC m=+1989.319139324" watchObservedRunningTime="2026-09-17 02:10:44.484696774 +0000 UTC m=+1989.319749143" Sep 17 02:10:48.177328 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:48.177293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:48.226914 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:48.226889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:10:48.774158 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:48.774127 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:10:55.477394 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:10:55.477362 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:11:05.818038 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:05.817991 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:06.664725 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:06.664679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:07.374836 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:07.374796 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:08.232696 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:08.232656 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:08.774771 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:08.774727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:09.442811 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:09.442773 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:09.903432 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:09.903401 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:10.246391 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.246359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:10.648893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.648815 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd"] Sep 17 02:11:10.652323 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.652307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:10.654481 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.654457 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:11:10.654609 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.654462 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:11:10.658611 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.658589 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd"] Sep 17 02:11:10.728365 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.728334 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd536-3a9b-4c51-82d1-aab04673b5c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd\" (UID: \"417bd536-3a9b-4c51-82d1-aab04673b5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:10.829804 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.829769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd536-3a9b-4c51-82d1-aab04673b5c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd\" (UID: \"417bd536-3a9b-4c51-82d1-aab04673b5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:10.830538 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.830516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd536-3a9b-4c51-82d1-aab04673b5c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd\" (UID: \"417bd536-3a9b-4c51-82d1-aab04673b5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:10.963621 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:10.963599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:11.135363 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:11.135334 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:11.232947 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:11.232887 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd"] Sep 17 02:11:11.236077 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:11:11.236051 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417bd536_3a9b_4c51_82d1_aab04673b5c9.slice/crio-9004d3e10811f7609bdae7e2ecd7bd8094da673ea37636f1bdefbb0ad37a65fb WatchSource:0}: Error finding container 9004d3e10811f7609bdae7e2ecd7bd8094da673ea37636f1bdefbb0ad37a65fb: Status 404 returned error can't find the container with id 9004d3e10811f7609bdae7e2ecd7bd8094da673ea37636f1bdefbb0ad37a65fb Sep 17 02:11:11.623191 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:11.623120 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" event={"ID":"417bd536-3a9b-4c51-82d1-aab04673b5c9","Type":"ContainerStarted","Data":"0ae225f7417cb87e5c8cf4d0066d3f8c0c1600c56232a827c92dd3e4778f8a46"} Sep 17 02:11:11.623191 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:11.623172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" event={"ID":"417bd536-3a9b-4c51-82d1-aab04673b5c9","Type":"ContainerStarted","Data":"9004d3e10811f7609bdae7e2ecd7bd8094da673ea37636f1bdefbb0ad37a65fb"} Sep 17 02:11:11.639086 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:11.639044 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" podStartSLOduration=1.639030826 podStartE2EDuration="1.639030826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:11:11.637919877 +0000 UTC m=+2016.472972210" watchObservedRunningTime="2026-09-17 02:11:11.639030826 +0000 UTC m=+2016.474083156" Sep 17 02:11:11.818949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:11.818919 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:12.533408 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:12.533361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:12.630085 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:12.630055 2582 generic.go:356] "Generic (PLEG): container finished" podID="417bd536-3a9b-4c51-82d1-aab04673b5c9" containerID="0ae225f7417cb87e5c8cf4d0066d3f8c0c1600c56232a827c92dd3e4778f8a46" exitCode=0 Sep 17 02:11:12.630249 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:12.630127 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" event={"ID":"417bd536-3a9b-4c51-82d1-aab04673b5c9","Type":"ContainerDied","Data":"0ae225f7417cb87e5c8cf4d0066d3f8c0c1600c56232a827c92dd3e4778f8a46"} Sep 17 02:11:13.354703 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:13.354659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:13.783042 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:13.783005 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:13.853955 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:13.853917 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd536-3a9b-4c51-82d1-aab04673b5c9-maas-api-service-ca\") pod \"417bd536-3a9b-4c51-82d1-aab04673b5c9\" (UID: \"417bd536-3a9b-4c51-82d1-aab04673b5c9\") " Sep 17 02:11:13.854345 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:13.854321 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417bd536-3a9b-4c51-82d1-aab04673b5c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "417bd536-3a9b-4c51-82d1-aab04673b5c9" (UID: "417bd536-3a9b-4c51-82d1-aab04673b5c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:13.914863 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:13.914744 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:13.955482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:13.955440 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd536-3a9b-4c51-82d1-aab04673b5c9-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:14.649530 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:14.649485 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" event={"ID":"417bd536-3a9b-4c51-82d1-aab04673b5c9","Type":"ContainerDied","Data":"9004d3e10811f7609bdae7e2ecd7bd8094da673ea37636f1bdefbb0ad37a65fb"} Sep 17 02:11:14.649763 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:14.649559 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9004d3e10811f7609bdae7e2ecd7bd8094da673ea37636f1bdefbb0ad37a65fb" Sep 17 02:11:14.649763 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:14.649497 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd" Sep 17 02:11:14.668091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:14.668052 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:21.141437 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:21.141398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:21.937123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:21.937087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:21.966977 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:21.966936 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv"] Sep 17 02:11:21.967236 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:21.967183 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" podUID="7bd7d6f4-7b26-49b1-8856-07055f476554" containerName="payload-pre-processing" containerID="cri-o://386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064" gracePeriod=30 Sep 17 02:11:22.000557 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.000525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:22.101867 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.101832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:22.130278 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.130199 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd"] Sep 17 02:11:22.131973 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.131945 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-557e7f-hvfxd"] Sep 17 02:11:22.158362 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.158322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:22.247028 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.246994 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:11:22.335291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.335255 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2v6b\" (UniqueName: \"kubernetes.io/projected/7bd7d6f4-7b26-49b1-8856-07055f476554-kube-api-access-v2v6b\") pod \"7bd7d6f4-7b26-49b1-8856-07055f476554\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " Sep 17 02:11:22.335457 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.335371 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7d6f4-7b26-49b1-8856-07055f476554-plugins-config-volume\") pod \"7bd7d6f4-7b26-49b1-8856-07055f476554\" (UID: \"7bd7d6f4-7b26-49b1-8856-07055f476554\") " Sep 17 02:11:22.335742 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.335717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd7d6f4-7b26-49b1-8856-07055f476554-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bd7d6f4-7b26-49b1-8856-07055f476554" (UID: "7bd7d6f4-7b26-49b1-8856-07055f476554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:22.337469 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.337443 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd7d6f4-7b26-49b1-8856-07055f476554-kube-api-access-v2v6b" (OuterVolumeSpecName: "kube-api-access-v2v6b") pod "7bd7d6f4-7b26-49b1-8856-07055f476554" (UID: "7bd7d6f4-7b26-49b1-8856-07055f476554"). InnerVolumeSpecName "kube-api-access-v2v6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:11:22.436638 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.436601 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2v6b\" (UniqueName: \"kubernetes.io/projected/7bd7d6f4-7b26-49b1-8856-07055f476554-kube-api-access-v2v6b\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:22.436638 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.436632 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7d6f4-7b26-49b1-8856-07055f476554-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:22.544500 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.544387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:22.644890 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.644846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:22.695262 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.695233 2582 generic.go:356] "Generic (PLEG): container finished" podID="7bd7d6f4-7b26-49b1-8856-07055f476554" containerID="386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064" exitCode=0 Sep 17 02:11:22.695428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.695302 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" event={"ID":"7bd7d6f4-7b26-49b1-8856-07055f476554","Type":"ContainerDied","Data":"386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064"} Sep 17 02:11:22.695428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.695325 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" Sep 17 02:11:22.695428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.695335 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv" event={"ID":"7bd7d6f4-7b26-49b1-8856-07055f476554","Type":"ContainerDied","Data":"4eea115387512f818b28e358a257e6c0aa9da922cbcc5ef2211dd9163e6f771f"} Sep 17 02:11:22.695428 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.695358 2582 scope.go:117] "RemoveContainer" containerID="386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064" Sep 17 02:11:22.704616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.704596 2582 scope.go:117] "RemoveContainer" containerID="386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064" Sep 17 02:11:22.704898 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:11:22.704878 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064\": container with ID starting with 386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064 not found: ID does not exist" containerID="386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064" Sep 17 02:11:22.704949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.704909 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064"} err="failed to get container status \"386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064\": rpc error: code = NotFound desc = could not find container \"386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064\": container with ID starting with 386ccd6931d6dc0523df5f721c53aba7d9f67e005b1ae1492b578afedc85c064 not found: ID does not exist" Sep 17 02:11:22.715546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.715523 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv"] Sep 17 02:11:22.721061 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:22.721038 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-557e7f-dbc4b54-lv5mv"] Sep 17 02:11:23.328823 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:23.328781 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:23.680044 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:23.679970 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417bd536-3a9b-4c51-82d1-aab04673b5c9" path="/var/lib/kubelet/pods/417bd536-3a9b-4c51-82d1-aab04673b5c9/volumes" Sep 17 02:11:23.680390 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:23.680375 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd7d6f4-7b26-49b1-8856-07055f476554" path="/var/lib/kubelet/pods/7bd7d6f4-7b26-49b1-8856-07055f476554/volumes" Sep 17 02:11:24.107770 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:24.107739 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:24.943136 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:24.943106 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:25.626675 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:25.626642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:26.576959 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:26.576925 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:11:28.115494 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:28.115459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:28.221083 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:28.221054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:28.405848 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:28.405762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:32.239724 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.239683 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf"] Sep 17 02:11:32.240335 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.240317 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd7d6f4-7b26-49b1-8856-07055f476554" containerName="payload-pre-processing" Sep 17 02:11:32.240396 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.240338 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd7d6f4-7b26-49b1-8856-07055f476554" containerName="payload-pre-processing" Sep 17 02:11:32.240396 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.240354 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417bd536-3a9b-4c51-82d1-aab04673b5c9" containerName="revoke-keys" Sep 17 02:11:32.240396 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.240363 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="417bd536-3a9b-4c51-82d1-aab04673b5c9" containerName="revoke-keys" Sep 17 02:11:32.240503 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.240490 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd7d6f4-7b26-49b1-8856-07055f476554" containerName="payload-pre-processing" Sep 17 02:11:32.240503 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.240501 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="417bd536-3a9b-4c51-82d1-aab04673b5c9" containerName="revoke-keys" Sep 17 02:11:32.245382 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.245360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:32.249381 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.248984 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:11:32.249381 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.248993 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:11:32.249381 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.249307 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf"] Sep 17 02:11:32.327526 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.327492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cab4a3-b1c7-40a3-9a94-309af44c0a49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf\" (UID: \"10cab4a3-b1c7-40a3-9a94-309af44c0a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:32.428306 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.428268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cab4a3-b1c7-40a3-9a94-309af44c0a49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf\" (UID: \"10cab4a3-b1c7-40a3-9a94-309af44c0a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:32.429096 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.429068 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cab4a3-b1c7-40a3-9a94-309af44c0a49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf\" (UID: \"10cab4a3-b1c7-40a3-9a94-309af44c0a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:32.529638 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.529537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:32.558343 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.558304 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:32.828650 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:32.828623 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf"] Sep 17 02:11:32.830747 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:11:32.830717 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cab4a3_b1c7_40a3_9a94_309af44c0a49.slice/crio-05af97343c715ca101f42b64a10855a17f90c2753bd71a9a9f51498d8b5f128d WatchSource:0}: Error finding container 05af97343c715ca101f42b64a10855a17f90c2753bd71a9a9f51498d8b5f128d: Status 404 returned error can't find the container with id 05af97343c715ca101f42b64a10855a17f90c2753bd71a9a9f51498d8b5f128d Sep 17 02:11:33.213179 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:33.213146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:33.762546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:33.762513 2582 generic.go:356] "Generic (PLEG): container finished" podID="10cab4a3-b1c7-40a3-9a94-309af44c0a49" containerID="8de0ccbde69ecb28d91b74965d7f5daf78f6b71f2dd5a05e8dd085fec60f1033" exitCode=0 Sep 17 02:11:33.762910 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:33.762580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" event={"ID":"10cab4a3-b1c7-40a3-9a94-309af44c0a49","Type":"ContainerDied","Data":"8de0ccbde69ecb28d91b74965d7f5daf78f6b71f2dd5a05e8dd085fec60f1033"} Sep 17 02:11:33.762910 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:33.762617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" event={"ID":"10cab4a3-b1c7-40a3-9a94-309af44c0a49","Type":"ContainerStarted","Data":"05af97343c715ca101f42b64a10855a17f90c2753bd71a9a9f51498d8b5f128d"} Sep 17 02:11:33.772367 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:33.772347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:34.545858 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:34.545824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:34.898817 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:34.898793 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:34.950662 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:34.950632 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cab4a3-b1c7-40a3-9a94-309af44c0a49-maas-api-service-ca\") pod \"10cab4a3-b1c7-40a3-9a94-309af44c0a49\" (UID: \"10cab4a3-b1c7-40a3-9a94-309af44c0a49\") " Sep 17 02:11:34.950969 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:34.950947 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cab4a3-b1c7-40a3-9a94-309af44c0a49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10cab4a3-b1c7-40a3-9a94-309af44c0a49" (UID: "10cab4a3-b1c7-40a3-9a94-309af44c0a49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:35.052409 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:35.052377 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cab4a3-b1c7-40a3-9a94-309af44c0a49-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:35.222806 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:35.222771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:35.775109 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:35.775079 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" Sep 17 02:11:35.775267 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:35.775107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf" event={"ID":"10cab4a3-b1c7-40a3-9a94-309af44c0a49","Type":"ContainerDied","Data":"05af97343c715ca101f42b64a10855a17f90c2753bd71a9a9f51498d8b5f128d"} Sep 17 02:11:35.775267 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:35.775153 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05af97343c715ca101f42b64a10855a17f90c2753bd71a9a9f51498d8b5f128d" Sep 17 02:11:36.028611 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:36.028579 2582 scope.go:117] "RemoveContainer" containerID="55b1281dbd2096ea1671779898019fd9713a42aa35ca3a6fea56fc74508099ee" Sep 17 02:11:36.057865 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:36.057835 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:38.119612 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:38.119567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:42.539706 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:42.539657 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:45.968541 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:45.968509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:11:48.249379 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.249340 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:48.279612 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.279566 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb"] Sep 17 02:11:48.279875 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.279840 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" podUID="6dfc0e30-9a18-4796-a358-e60688b6bbef" containerName="payload-processing" containerID="cri-o://596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146" gracePeriod=30 Sep 17 02:11:48.289638 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.289435 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7"] Sep 17 02:11:48.289733 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.289659 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" podUID="91374ce7-81b0-4db5-8bb4-49e10cd367de" containerName="payload-pre-processing" containerID="cri-o://534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74" gracePeriod=30 Sep 17 02:11:48.445504 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.445457 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf"] Sep 17 02:11:48.447291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.447252 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0d7c34-stscf"] Sep 17 02:11:48.563790 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.563767 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:11:48.566682 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.566662 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:11:48.678713 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.678686 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91374ce7-81b0-4db5-8bb4-49e10cd367de-plugins-config-volume\") pod \"91374ce7-81b0-4db5-8bb4-49e10cd367de\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " Sep 17 02:11:48.678893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.678732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc7qn\" (UniqueName: \"kubernetes.io/projected/91374ce7-81b0-4db5-8bb4-49e10cd367de-kube-api-access-vc7qn\") pod \"91374ce7-81b0-4db5-8bb4-49e10cd367de\" (UID: \"91374ce7-81b0-4db5-8bb4-49e10cd367de\") " Sep 17 02:11:48.678893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.678809 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfc0e30-9a18-4796-a358-e60688b6bbef-plugins-config-volume\") pod \"6dfc0e30-9a18-4796-a358-e60688b6bbef\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " Sep 17 02:11:48.678893 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.678830 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6r67\" (UniqueName: \"kubernetes.io/projected/6dfc0e30-9a18-4796-a358-e60688b6bbef-kube-api-access-t6r67\") pod \"6dfc0e30-9a18-4796-a358-e60688b6bbef\" (UID: \"6dfc0e30-9a18-4796-a358-e60688b6bbef\") " Sep 17 02:11:48.679086 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.679056 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91374ce7-81b0-4db5-8bb4-49e10cd367de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91374ce7-81b0-4db5-8bb4-49e10cd367de" (UID: "91374ce7-81b0-4db5-8bb4-49e10cd367de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:48.679195 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.679172 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfc0e30-9a18-4796-a358-e60688b6bbef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dfc0e30-9a18-4796-a358-e60688b6bbef" (UID: "6dfc0e30-9a18-4796-a358-e60688b6bbef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:11:48.680820 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.680790 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91374ce7-81b0-4db5-8bb4-49e10cd367de-kube-api-access-vc7qn" (OuterVolumeSpecName: "kube-api-access-vc7qn") pod "91374ce7-81b0-4db5-8bb4-49e10cd367de" (UID: "91374ce7-81b0-4db5-8bb4-49e10cd367de"). InnerVolumeSpecName "kube-api-access-vc7qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:11:48.680820 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.680810 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfc0e30-9a18-4796-a358-e60688b6bbef-kube-api-access-t6r67" (OuterVolumeSpecName: "kube-api-access-t6r67") pod "6dfc0e30-9a18-4796-a358-e60688b6bbef" (UID: "6dfc0e30-9a18-4796-a358-e60688b6bbef"). InnerVolumeSpecName "kube-api-access-t6r67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:11:48.779854 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.779804 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfc0e30-9a18-4796-a358-e60688b6bbef-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.779854 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.779845 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6r67\" (UniqueName: \"kubernetes.io/projected/6dfc0e30-9a18-4796-a358-e60688b6bbef-kube-api-access-t6r67\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.779854 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.779860 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91374ce7-81b0-4db5-8bb4-49e10cd367de-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.780158 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.779872 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7qn\" (UniqueName: \"kubernetes.io/projected/91374ce7-81b0-4db5-8bb4-49e10cd367de-kube-api-access-vc7qn\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:11:48.844905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.844808 2582 generic.go:356] "Generic (PLEG): container finished" podID="6dfc0e30-9a18-4796-a358-e60688b6bbef" containerID="596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146" exitCode=0 Sep 17 02:11:48.844905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.844852 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" Sep 17 02:11:48.845135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.844901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" event={"ID":"6dfc0e30-9a18-4796-a358-e60688b6bbef","Type":"ContainerDied","Data":"596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146"} Sep 17 02:11:48.845135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.844961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb" event={"ID":"6dfc0e30-9a18-4796-a358-e60688b6bbef","Type":"ContainerDied","Data":"8aa1bf157da4c05083af0e8b113d4445acaed56cea303a50b5d826782635e51e"} Sep 17 02:11:48.845135 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.844991 2582 scope.go:117] "RemoveContainer" containerID="596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146" Sep 17 02:11:48.846555 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.846530 2582 generic.go:356] "Generic (PLEG): container finished" podID="91374ce7-81b0-4db5-8bb4-49e10cd367de" containerID="534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74" exitCode=0 Sep 17 02:11:48.846700 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.846572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" event={"ID":"91374ce7-81b0-4db5-8bb4-49e10cd367de","Type":"ContainerDied","Data":"534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74"} Sep 17 02:11:48.846700 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.846584 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" Sep 17 02:11:48.846700 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.846613 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7" event={"ID":"91374ce7-81b0-4db5-8bb4-49e10cd367de","Type":"ContainerDied","Data":"50a8ae88366ca0662fb38972d2f1da993cb88a99b98708223c32d64a7d8ddf42"} Sep 17 02:11:48.856423 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.856403 2582 scope.go:117] "RemoveContainer" containerID="596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146" Sep 17 02:11:48.856794 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:11:48.856769 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146\": container with ID starting with 596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146 not found: ID does not exist" containerID="596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146" Sep 17 02:11:48.856894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.856804 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146"} err="failed to get container status \"596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146\": rpc error: code = NotFound desc = could not find container \"596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146\": container with ID starting with 596d9ce5e8aaff83f96a343e0bfc2a85da40a8e8f506959f189beb95c1a8c146 not found: ID does not exist" Sep 17 02:11:48.856894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.856827 2582 scope.go:117] "RemoveContainer" containerID="534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74" Sep 17 02:11:48.866443 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.866417 2582 scope.go:117] "RemoveContainer" containerID="534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74" Sep 17 02:11:48.866790 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:11:48.866767 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74\": container with ID starting with 534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74 not found: ID does not exist" containerID="534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74" Sep 17 02:11:48.866866 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.866797 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74"} err="failed to get container status \"534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74\": rpc error: code = NotFound desc = could not find container \"534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74\": container with ID starting with 534fa63f01904dbf9bc3bee417aedf61a6cab2d867d009f715bf7e3a7bd43e74 not found: ID does not exist" Sep 17 02:11:48.868593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.868573 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb"] Sep 17 02:11:48.871610 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.871588 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0d7c34-8665f4db95-sgbrb"] Sep 17 02:11:48.882485 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.882462 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7"] Sep 17 02:11:48.888755 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:48.888733 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0d7c34-64cf9b44bb-lzxm7"] Sep 17 02:11:49.525865 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:49.525836 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:49.679680 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:49.679649 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10cab4a3-b1c7-40a3-9a94-309af44c0a49" path="/var/lib/kubelet/pods/10cab4a3-b1c7-40a3-9a94-309af44c0a49/volumes" Sep 17 02:11:49.680031 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:49.680016 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dfc0e30-9a18-4796-a358-e60688b6bbef" path="/var/lib/kubelet/pods/6dfc0e30-9a18-4796-a358-e60688b6bbef/volumes" Sep 17 02:11:49.680376 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:49.680362 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91374ce7-81b0-4db5-8bb4-49e10cd367de" path="/var/lib/kubelet/pods/91374ce7-81b0-4db5-8bb4-49e10cd367de/volumes" Sep 17 02:11:50.217558 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:50.217520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:51.048726 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:51.048691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:53.838291 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:53.838260 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms"] Sep 17 02:11:53.935657 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:53.935621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:11:58.773063 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:11:58.772985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:12:08.857519 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:08.857470 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" podUID="0905a014-0f8c-4171-a0e2-2c5c426b11b3" containerName="main" containerID="cri-o://c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1" gracePeriod=45 Sep 17 02:12:09.106668 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.106640 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:12:09.247522 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247487 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/0905a014-0f8c-4171-a0e2-2c5c426b11b3-kube-api-access-nzw8d\") pod \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " Sep 17 02:12:09.247709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247553 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-home\") pod \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " Sep 17 02:12:09.247709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247627 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tmp-dir\") pod \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " Sep 17 02:12:09.247709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247650 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tls-certs\") pod \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " Sep 17 02:12:09.247709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247667 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-model-cache\") pod \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " Sep 17 02:12:09.247709 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247702 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-dshm\") pod \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\" (UID: \"0905a014-0f8c-4171-a0e2-2c5c426b11b3\") " Sep 17 02:12:09.247984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247809 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-home" (OuterVolumeSpecName: "home") pod "0905a014-0f8c-4171-a0e2-2c5c426b11b3" (UID: "0905a014-0f8c-4171-a0e2-2c5c426b11b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:12:09.247984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247847 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0905a014-0f8c-4171-a0e2-2c5c426b11b3" (UID: "0905a014-0f8c-4171-a0e2-2c5c426b11b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:12:09.247984 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247972 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:09.248144 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.247980 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0905a014-0f8c-4171-a0e2-2c5c426b11b3" (UID: "0905a014-0f8c-4171-a0e2-2c5c426b11b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:12:09.248144 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.248053 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:09.249967 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.249937 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0905a014-0f8c-4171-a0e2-2c5c426b11b3" (UID: "0905a014-0f8c-4171-a0e2-2c5c426b11b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:12:09.250086 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.249966 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-dshm" (OuterVolumeSpecName: "dshm") pod "0905a014-0f8c-4171-a0e2-2c5c426b11b3" (UID: "0905a014-0f8c-4171-a0e2-2c5c426b11b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:12:09.250086 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.249995 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0905a014-0f8c-4171-a0e2-2c5c426b11b3-kube-api-access-nzw8d" (OuterVolumeSpecName: "kube-api-access-nzw8d") pod "0905a014-0f8c-4171-a0e2-2c5c426b11b3" (UID: "0905a014-0f8c-4171-a0e2-2c5c426b11b3"). InnerVolumeSpecName "kube-api-access-nzw8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:12:09.348843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.348799 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905a014-0f8c-4171-a0e2-2c5c426b11b3-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:09.348843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.348836 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:09.348843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.348845 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905a014-0f8c-4171-a0e2-2c5c426b11b3-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:09.348843 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.348854 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzw8d\" (UniqueName: \"kubernetes.io/projected/0905a014-0f8c-4171-a0e2-2c5c426b11b3-kube-api-access-nzw8d\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:09.965546 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.965513 2582 generic.go:356] "Generic (PLEG): container finished" podID="0905a014-0f8c-4171-a0e2-2c5c426b11b3" containerID="c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1" exitCode=0 Sep 17 02:12:09.966029 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.965557 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" Sep 17 02:12:09.966029 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.965593 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" event={"ID":"0905a014-0f8c-4171-a0e2-2c5c426b11b3","Type":"ContainerDied","Data":"c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1"} Sep 17 02:12:09.966029 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.965647 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms" event={"ID":"0905a014-0f8c-4171-a0e2-2c5c426b11b3","Type":"ContainerDied","Data":"ff71d184442b636fe84886a3d1691bff1f131ff744fb112a4058344c9c866caf"} Sep 17 02:12:09.966029 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.965664 2582 scope.go:117] "RemoveContainer" containerID="c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1" Sep 17 02:12:09.974720 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.974699 2582 scope.go:117] "RemoveContainer" containerID="c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1" Sep 17 02:12:09.975062 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:12:09.975042 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1\": container with ID starting with c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1 not found: ID does not exist" containerID="c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1" Sep 17 02:12:09.975127 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.975077 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1"} err="failed to get container status \"c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1\": rpc error: code = NotFound desc = could not find container \"c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1\": container with ID starting with c2113f5319513cc254d24c6cf6b68e1e7a4bc3e029b0ff1f192bfbfe5ba155a1 not found: ID does not exist" Sep 17 02:12:09.980482 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.980457 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms"] Sep 17 02:12:09.984376 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:09.984350 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0d7c34/ipp-route-0d7c34-kserve-f998bdfcf-zhvms"] Sep 17 02:12:11.678805 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:11.678772 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0905a014-0f8c-4171-a0e2-2c5c426b11b3" path="/var/lib/kubelet/pods/0905a014-0f8c-4171-a0e2-2c5c426b11b3/volumes" Sep 17 02:12:29.978615 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:29.978580 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:31.064844 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:31.064809 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:36.527764 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:36.527705 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:36.591149 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:36.591110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:36.830626 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:36.830511 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:37.190625 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.190534 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92"] Sep 17 02:12:37.191004 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.190984 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91374ce7-81b0-4db5-8bb4-49e10cd367de" containerName="payload-pre-processing" Sep 17 02:12:37.191091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191007 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91374ce7-81b0-4db5-8bb4-49e10cd367de" containerName="payload-pre-processing" Sep 17 02:12:37.191091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191036 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10cab4a3-b1c7-40a3-9a94-309af44c0a49" containerName="revoke-keys" Sep 17 02:12:37.191091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191045 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="10cab4a3-b1c7-40a3-9a94-309af44c0a49" containerName="revoke-keys" Sep 17 02:12:37.191091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191074 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dfc0e30-9a18-4796-a358-e60688b6bbef" containerName="payload-processing" Sep 17 02:12:37.191091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191083 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dfc0e30-9a18-4796-a358-e60688b6bbef" containerName="payload-processing" Sep 17 02:12:37.191398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191097 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0905a014-0f8c-4171-a0e2-2c5c426b11b3" containerName="main" Sep 17 02:12:37.191398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191105 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0905a014-0f8c-4171-a0e2-2c5c426b11b3" containerName="main" Sep 17 02:12:37.191398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191193 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="10cab4a3-b1c7-40a3-9a94-309af44c0a49" containerName="revoke-keys" Sep 17 02:12:37.191398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191204 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dfc0e30-9a18-4796-a358-e60688b6bbef" containerName="payload-processing" Sep 17 02:12:37.191398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191235 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0905a014-0f8c-4171-a0e2-2c5c426b11b3" containerName="main" Sep 17 02:12:37.191398 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.191245 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91374ce7-81b0-4db5-8bb4-49e10cd367de" containerName="payload-pre-processing" Sep 17 02:12:37.195551 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.195529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:37.198693 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.198648 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:12:37.198837 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.198648 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:12:37.201333 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.201306 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92"] Sep 17 02:12:37.277586 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.277522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e099bb-3883-4245-b4a8-57d0d88bd770-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92\" (UID: \"c9e099bb-3883-4245-b4a8-57d0d88bd770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:37.378490 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.378455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e099bb-3883-4245-b4a8-57d0d88bd770-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92\" (UID: \"c9e099bb-3883-4245-b4a8-57d0d88bd770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:37.379129 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.379100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e099bb-3883-4245-b4a8-57d0d88bd770-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92\" (UID: \"c9e099bb-3883-4245-b4a8-57d0d88bd770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:37.473427 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.473383 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:37.507400 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.507374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:37.776372 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:37.776349 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92"] Sep 17 02:12:37.778866 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:12:37.778842 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e099bb_3883_4245_b4a8_57d0d88bd770.slice/crio-b32aea95b499a2000444c59e924fb7f83005ca6c99bd3925fda18cda62ab6738 WatchSource:0}: Error finding container b32aea95b499a2000444c59e924fb7f83005ca6c99bd3925fda18cda62ab6738: Status 404 returned error can't find the container with id b32aea95b499a2000444c59e924fb7f83005ca6c99bd3925fda18cda62ab6738 Sep 17 02:12:38.119128 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:38.119093 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" event={"ID":"c9e099bb-3883-4245-b4a8-57d0d88bd770","Type":"ContainerStarted","Data":"b32aea95b499a2000444c59e924fb7f83005ca6c99bd3925fda18cda62ab6738"} Sep 17 02:12:38.175907 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:38.175877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:38.938910 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:38.938814 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:39.126721 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:39.126681 2582 generic.go:356] "Generic (PLEG): container finished" podID="c9e099bb-3883-4245-b4a8-57d0d88bd770" containerID="ed430dc6abce6244cc0f1bd58118addef3ef393fcb4c55bdedc66fdf31ad3425" exitCode=0 Sep 17 02:12:39.126894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:39.126727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" event={"ID":"c9e099bb-3883-4245-b4a8-57d0d88bd770","Type":"ContainerDied","Data":"ed430dc6abce6244cc0f1bd58118addef3ef393fcb4c55bdedc66fdf31ad3425"} Sep 17 02:12:39.630505 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:39.630463 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:40.176088 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:40.176051 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:40.256548 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:40.256528 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:40.301969 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:40.301944 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e099bb-3883-4245-b4a8-57d0d88bd770-maas-api-service-ca\") pod \"c9e099bb-3883-4245-b4a8-57d0d88bd770\" (UID: \"c9e099bb-3883-4245-b4a8-57d0d88bd770\") " Sep 17 02:12:40.302277 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:40.302256 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e099bb-3883-4245-b4a8-57d0d88bd770-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9e099bb-3883-4245-b4a8-57d0d88bd770" (UID: "c9e099bb-3883-4245-b4a8-57d0d88bd770"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:12:40.403035 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:40.402985 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e099bb-3883-4245-b4a8-57d0d88bd770-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:41.138343 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:41.138313 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" event={"ID":"c9e099bb-3883-4245-b4a8-57d0d88bd770","Type":"ContainerDied","Data":"b32aea95b499a2000444c59e924fb7f83005ca6c99bd3925fda18cda62ab6738"} Sep 17 02:12:41.138343 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:41.138342 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92" Sep 17 02:12:41.138541 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:41.138363 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b32aea95b499a2000444c59e924fb7f83005ca6c99bd3925fda18cda62ab6738" Sep 17 02:12:47.755594 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:47.755555 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:47.792904 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:47.792879 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc"] Sep 17 02:12:47.793287 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:47.793253 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" podUID="31a4e869-0117-437e-9815-9738bd2edc77" containerName="payload-pre-processing" containerID="cri-o://05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2" gracePeriod=30 Sep 17 02:12:47.857591 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:47.857563 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:47.974129 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:47.974096 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:48.051000 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.050973 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92"] Sep 17 02:12:48.053954 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.053931 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:12:48.057413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.057393 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee767-8ps92"] Sep 17 02:12:48.090023 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.089999 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:48.166668 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.166644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a4e869-0117-437e-9815-9738bd2edc77-plugins-config-volume\") pod \"31a4e869-0117-437e-9815-9738bd2edc77\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " Sep 17 02:12:48.166785 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.166695 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/31a4e869-0117-437e-9815-9738bd2edc77-kube-api-access-mz56b\") pod \"31a4e869-0117-437e-9815-9738bd2edc77\" (UID: \"31a4e869-0117-437e-9815-9738bd2edc77\") " Sep 17 02:12:48.167008 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.166984 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a4e869-0117-437e-9815-9738bd2edc77-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31a4e869-0117-437e-9815-9738bd2edc77" (UID: "31a4e869-0117-437e-9815-9738bd2edc77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:12:48.168785 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.168764 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a4e869-0117-437e-9815-9738bd2edc77-kube-api-access-mz56b" (OuterVolumeSpecName: "kube-api-access-mz56b") pod "31a4e869-0117-437e-9815-9738bd2edc77" (UID: "31a4e869-0117-437e-9815-9738bd2edc77"). InnerVolumeSpecName "kube-api-access-mz56b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:12:48.173847 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.173822 2582 generic.go:356] "Generic (PLEG): container finished" podID="31a4e869-0117-437e-9815-9738bd2edc77" containerID="05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2" exitCode=0 Sep 17 02:12:48.173946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.173861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" event={"ID":"31a4e869-0117-437e-9815-9738bd2edc77","Type":"ContainerDied","Data":"05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2"} Sep 17 02:12:48.173946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.173925 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" event={"ID":"31a4e869-0117-437e-9815-9738bd2edc77","Type":"ContainerDied","Data":"8c813eda7435b7fa9eea2fd783493cd012f52a8ee7ffb2ea86a8a0632f48a1a2"} Sep 17 02:12:48.173946 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.173868 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc" Sep 17 02:12:48.174118 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.173959 2582 scope.go:117] "RemoveContainer" containerID="05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2" Sep 17 02:12:48.182518 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.182502 2582 scope.go:117] "RemoveContainer" containerID="05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2" Sep 17 02:12:48.182767 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:12:48.182746 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2\": container with ID starting with 05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2 not found: ID does not exist" containerID="05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2" Sep 17 02:12:48.182824 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.182775 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2"} err="failed to get container status \"05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2\": rpc error: code = NotFound desc = could not find container \"05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2\": container with ID starting with 05c94fc2abe383787b330e82ebdc3e7e67a7b67bceea87b2f67a7cdd769aa4d2 not found: ID does not exist" Sep 17 02:12:48.194334 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.194315 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc"] Sep 17 02:12:48.198175 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.198154 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee767-5f6c8d6b98-rcjbc"] Sep 17 02:12:48.268095 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.268070 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a4e869-0117-437e-9815-9738bd2edc77-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:48.268095 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.268091 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz56b\" (UniqueName: \"kubernetes.io/projected/31a4e869-0117-437e-9815-9738bd2edc77-kube-api-access-mz56b\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:48.442783 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.442705 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:48.532192 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:48.532167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:49.196153 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:49.196119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:49.679544 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:49.679514 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a4e869-0117-437e-9815-9738bd2edc77" path="/var/lib/kubelet/pods/31a4e869-0117-437e-9815-9738bd2edc77/volumes" Sep 17 02:12:49.679846 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:49.679833 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9e099bb-3883-4245-b4a8-57d0d88bd770" path="/var/lib/kubelet/pods/c9e099bb-3883-4245-b4a8-57d0d88bd770/volumes" Sep 17 02:12:50.014413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:50.014376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:50.977986 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:50.977953 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:12:54.196222 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.196184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:54.262380 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.262347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:54.474727 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.474688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:54.825805 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.825734 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg"] Sep 17 02:12:54.826123 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.826109 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e099bb-3883-4245-b4a8-57d0d88bd770" containerName="revoke-keys" Sep 17 02:12:54.826167 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.826125 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e099bb-3883-4245-b4a8-57d0d88bd770" containerName="revoke-keys" Sep 17 02:12:54.826167 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.826137 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a4e869-0117-437e-9815-9738bd2edc77" containerName="payload-pre-processing" Sep 17 02:12:54.826167 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.826142 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a4e869-0117-437e-9815-9738bd2edc77" containerName="payload-pre-processing" Sep 17 02:12:54.826292 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.826202 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e099bb-3883-4245-b4a8-57d0d88bd770" containerName="revoke-keys" Sep 17 02:12:54.826292 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.826225 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a4e869-0117-437e-9815-9738bd2edc77" containerName="payload-pre-processing" Sep 17 02:12:54.830518 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.830502 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:54.832710 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.832688 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:12:54.832846 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.832729 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:12:54.836401 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.836378 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg"] Sep 17 02:12:54.918589 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.918564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f621d8f-93b9-4db4-88df-6950274fabcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg\" (UID: \"7f621d8f-93b9-4db4-88df-6950274fabcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:54.956260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:54.956196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:55.019263 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:55.019237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f621d8f-93b9-4db4-88df-6950274fabcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg\" (UID: \"7f621d8f-93b9-4db4-88df-6950274fabcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:55.019800 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:55.019782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f621d8f-93b9-4db4-88df-6950274fabcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg\" (UID: \"7f621d8f-93b9-4db4-88df-6950274fabcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:55.143532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:55.143458 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:55.411036 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:55.411012 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg"] Sep 17 02:12:55.413040 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:12:55.413005 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f621d8f_93b9_4db4_88df_6950274fabcd.slice/crio-a76957b6511e8955a6ad7ad20c089f9879e21c777588426054910b40d9f9cea7 WatchSource:0}: Error finding container a76957b6511e8955a6ad7ad20c089f9879e21c777588426054910b40d9f9cea7: Status 404 returned error can't find the container with id a76957b6511e8955a6ad7ad20c089f9879e21c777588426054910b40d9f9cea7 Sep 17 02:12:55.919813 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:55.919774 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:56.224051 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:56.224019 2582 generic.go:356] "Generic (PLEG): container finished" podID="7f621d8f-93b9-4db4-88df-6950274fabcd" containerID="aba627e764fa5836f66dcb53b1e8b5a4e1d671c39b4171e872c06ed9991e45d3" exitCode=0 Sep 17 02:12:56.224389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:56.224112 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" event={"ID":"7f621d8f-93b9-4db4-88df-6950274fabcd","Type":"ContainerDied","Data":"aba627e764fa5836f66dcb53b1e8b5a4e1d671c39b4171e872c06ed9991e45d3"} Sep 17 02:12:56.224389 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:56.224193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" event={"ID":"7f621d8f-93b9-4db4-88df-6950274fabcd","Type":"ContainerStarted","Data":"a76957b6511e8955a6ad7ad20c089f9879e21c777588426054910b40d9f9cea7"} Sep 17 02:12:56.496250 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:56.496197 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:57.166633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:57.166594 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:12:57.357192 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:57.357167 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:57.437105 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:57.437020 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f621d8f-93b9-4db4-88df-6950274fabcd-maas-api-service-ca\") pod \"7f621d8f-93b9-4db4-88df-6950274fabcd\" (UID: \"7f621d8f-93b9-4db4-88df-6950274fabcd\") " Sep 17 02:12:57.437473 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:57.437449 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f621d8f-93b9-4db4-88df-6950274fabcd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f621d8f-93b9-4db4-88df-6950274fabcd" (UID: "7f621d8f-93b9-4db4-88df-6950274fabcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:12:57.538633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:57.538603 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f621d8f-93b9-4db4-88df-6950274fabcd-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:12:58.235045 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:58.235013 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" Sep 17 02:12:58.235228 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:58.235051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg" event={"ID":"7f621d8f-93b9-4db4-88df-6950274fabcd","Type":"ContainerDied","Data":"a76957b6511e8955a6ad7ad20c089f9879e21c777588426054910b40d9f9cea7"} Sep 17 02:12:58.235228 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:12:58.235097 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76957b6511e8955a6ad7ad20c089f9879e21c777588426054910b40d9f9cea7" Sep 17 02:13:05.320181 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.320146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:05.346693 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.346661 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm"] Sep 17 02:13:05.347037 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.346895 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" podUID="bcda171c-2cec-4968-88f4-3b4c725c7c4f" containerName="payload-processing" containerID="cri-o://f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41" gracePeriod=30 Sep 17 02:13:05.422676 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.422640 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:05.511425 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.511363 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg"] Sep 17 02:13:05.514202 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.514163 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-67a23a-f5gqg"] Sep 17 02:13:05.514356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.514195 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:05.570678 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.570600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:05.600874 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.600849 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:13:05.679645 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.679613 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f621d8f-93b9-4db4-88df-6950274fabcd" path="/var/lib/kubelet/pods/7f621d8f-93b9-4db4-88df-6950274fabcd/volumes" Sep 17 02:13:05.707618 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.707598 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcda171c-2cec-4968-88f4-3b4c725c7c4f-plugins-config-volume\") pod \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " Sep 17 02:13:05.707703 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.707647 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkjb8\" (UniqueName: \"kubernetes.io/projected/bcda171c-2cec-4968-88f4-3b4c725c7c4f-kube-api-access-dkjb8\") pod \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\" (UID: \"bcda171c-2cec-4968-88f4-3b4c725c7c4f\") " Sep 17 02:13:05.707972 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.707951 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcda171c-2cec-4968-88f4-3b4c725c7c4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bcda171c-2cec-4968-88f4-3b4c725c7c4f" (UID: "bcda171c-2cec-4968-88f4-3b4c725c7c4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:13:05.709795 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.709777 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda171c-2cec-4968-88f4-3b4c725c7c4f-kube-api-access-dkjb8" (OuterVolumeSpecName: "kube-api-access-dkjb8") pod "bcda171c-2cec-4968-88f4-3b4c725c7c4f" (UID: "bcda171c-2cec-4968-88f4-3b4c725c7c4f"). InnerVolumeSpecName "kube-api-access-dkjb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:13:05.808453 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.808425 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjb8\" (UniqueName: \"kubernetes.io/projected/bcda171c-2cec-4968-88f4-3b4c725c7c4f-kube-api-access-dkjb8\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:13:05.808453 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.808448 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcda171c-2cec-4968-88f4-3b4c725c7c4f-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:13:05.928676 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:05.928590 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:06.028530 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.028493 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:06.279958 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.279925 2582 generic.go:356] "Generic (PLEG): container finished" podID="bcda171c-2cec-4968-88f4-3b4c725c7c4f" containerID="f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41" exitCode=0 Sep 17 02:13:06.280113 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.279968 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" Sep 17 02:13:06.280113 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.279963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" event={"ID":"bcda171c-2cec-4968-88f4-3b4c725c7c4f","Type":"ContainerDied","Data":"f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41"} Sep 17 02:13:06.280113 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.280088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm" event={"ID":"bcda171c-2cec-4968-88f4-3b4c725c7c4f","Type":"ContainerDied","Data":"b837198bca6f717e070396d8a06c8d6f1227fa6d94c10d7b911df5878e3f87f1"} Sep 17 02:13:06.280293 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.280117 2582 scope.go:117] "RemoveContainer" containerID="f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41" Sep 17 02:13:06.289310 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.289292 2582 scope.go:117] "RemoveContainer" containerID="f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41" Sep 17 02:13:06.289568 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:13:06.289549 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41\": container with ID starting with f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41 not found: ID does not exist" containerID="f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41" Sep 17 02:13:06.289631 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.289577 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41"} err="failed to get container status \"f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41\": rpc error: code = NotFound desc = could not find container \"f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41\": container with ID starting with f71b07d09279cc43543a054c868e4f01b1f6828e9955b1b55eedf2fe60249d41 not found: ID does not exist" Sep 17 02:13:06.302745 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.302724 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm"] Sep 17 02:13:06.306472 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.306449 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-67a23a-864cbbbb65-9frnm"] Sep 17 02:13:06.521374 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:06.521332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:07.682386 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:07.680369 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcda171c-2cec-4968-88f4-3b4c725c7c4f" path="/var/lib/kubelet/pods/bcda171c-2cec-4968-88f4-3b4c725c7c4f/volumes" Sep 17 02:13:08.368792 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:08.368757 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:13:12.055642 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.055600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:12.114599 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.114568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:12.330168 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.330088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:12.687181 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687106 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj"] Sep 17 02:13:12.687484 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687470 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcda171c-2cec-4968-88f4-3b4c725c7c4f" containerName="payload-processing" Sep 17 02:13:12.687532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687486 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcda171c-2cec-4968-88f4-3b4c725c7c4f" containerName="payload-processing" Sep 17 02:13:12.687532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687508 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f621d8f-93b9-4db4-88df-6950274fabcd" containerName="revoke-keys" Sep 17 02:13:12.687532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687516 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f621d8f-93b9-4db4-88df-6950274fabcd" containerName="revoke-keys" Sep 17 02:13:12.687628 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687575 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcda171c-2cec-4968-88f4-3b4c725c7c4f" containerName="payload-processing" Sep 17 02:13:12.687628 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.687584 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f621d8f-93b9-4db4-88df-6950274fabcd" containerName="revoke-keys" Sep 17 02:13:12.690772 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.690757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:12.692980 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.692956 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-shqwh\"" Sep 17 02:13:12.693084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.692964 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:13:12.696680 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.696657 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj"] Sep 17 02:13:12.768768 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.768741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bc4649-6e03-4967-8222-461ccabe913a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj\" (UID: \"67bc4649-6e03-4967-8222-461ccabe913a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:12.869296 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.869275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bc4649-6e03-4967-8222-461ccabe913a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj\" (UID: \"67bc4649-6e03-4967-8222-461ccabe913a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:12.869816 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:12.869799 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bc4649-6e03-4967-8222-461ccabe913a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj\" (UID: \"67bc4649-6e03-4967-8222-461ccabe913a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:13.001811 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:13.001792 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:13.270343 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:13.270303 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj"] Sep 17 02:13:13.272619 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:13:13.272592 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bc4649_6e03_4967_8222_461ccabe913a.slice/crio-c0bc960b42607622e6dc844990b32f8b468ed95f1d4ae9c3007db0a9cdedb63a WatchSource:0}: Error finding container c0bc960b42607622e6dc844990b32f8b468ed95f1d4ae9c3007db0a9cdedb63a: Status 404 returned error can't find the container with id c0bc960b42607622e6dc844990b32f8b468ed95f1d4ae9c3007db0a9cdedb63a Sep 17 02:13:13.317650 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:13.317627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" event={"ID":"67bc4649-6e03-4967-8222-461ccabe913a","Type":"ContainerStarted","Data":"c0bc960b42607622e6dc844990b32f8b468ed95f1d4ae9c3007db0a9cdedb63a"} Sep 17 02:13:13.651470 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:13.651436 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:14.196867 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:14.196782 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:14.323974 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:14.323945 2582 generic.go:356] "Generic (PLEG): container finished" podID="67bc4649-6e03-4967-8222-461ccabe913a" containerID="cff5ff27dc368db8eaecc3595d724f7bd0e6d6e4cc2e8a6464aefc226312d417" exitCode=0 Sep 17 02:13:14.324382 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:14.324040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" event={"ID":"67bc4649-6e03-4967-8222-461ccabe913a","Type":"ContainerDied","Data":"cff5ff27dc368db8eaecc3595d724f7bd0e6d6e4cc2e8a6464aefc226312d417"} Sep 17 02:13:15.451237 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:15.451197 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:15.590451 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:15.590367 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bc4649-6e03-4967-8222-461ccabe913a-maas-api-service-ca\") pod \"67bc4649-6e03-4967-8222-461ccabe913a\" (UID: \"67bc4649-6e03-4967-8222-461ccabe913a\") " Sep 17 02:13:15.590704 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:15.590682 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bc4649-6e03-4967-8222-461ccabe913a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67bc4649-6e03-4967-8222-461ccabe913a" (UID: "67bc4649-6e03-4967-8222-461ccabe913a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:13:15.691599 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:15.691575 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bc4649-6e03-4967-8222-461ccabe913a-maas-api-service-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:13:16.097639 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:16.097607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:16.335895 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:16.335865 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" Sep 17 02:13:16.335895 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:16.335889 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj" event={"ID":"67bc4649-6e03-4967-8222-461ccabe913a","Type":"ContainerDied","Data":"c0bc960b42607622e6dc844990b32f8b468ed95f1d4ae9c3007db0a9cdedb63a"} Sep 17 02:13:16.336084 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:16.335937 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0bc960b42607622e6dc844990b32f8b468ed95f1d4ae9c3007db0a9cdedb63a" Sep 17 02:13:23.294616 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.294572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:23.323584 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.323554 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd"] Sep 17 02:13:23.324578 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.324361 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" podUID="bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" containerName="payload-pre-processing" containerID="cri-o://2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7" gracePeriod=30 Sep 17 02:13:23.391260 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.391229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:23.478575 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.478546 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj"] Sep 17 02:13:23.481410 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.481374 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:23.484237 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.484198 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f03337-rrwfj"] Sep 17 02:13:23.537601 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.537564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:23.587064 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.587043 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:13:23.679551 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.679523 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67bc4649-6e03-4967-8222-461ccabe913a" path="/var/lib/kubelet/pods/67bc4649-6e03-4967-8222-461ccabe913a/volumes" Sep 17 02:13:23.757498 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.757466 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-plugins-config-volume\") pod \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " Sep 17 02:13:23.757628 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.757554 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bgff\" (UniqueName: \"kubernetes.io/projected/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-kube-api-access-9bgff\") pod \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\" (UID: \"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7\") " Sep 17 02:13:23.757797 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.757775 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" (UID: "bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:13:23.759626 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.759605 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-kube-api-access-9bgff" (OuterVolumeSpecName: "kube-api-access-9bgff") pod "bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" (UID: "bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7"). InnerVolumeSpecName "kube-api-access-9bgff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:13:23.858329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.858262 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgff\" (UniqueName: \"kubernetes.io/projected/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-kube-api-access-9bgff\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:13:23.858329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.858291 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7-plugins-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:13:23.903956 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:23.903928 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:24.009524 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.009501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:24.381880 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.381852 2582 generic.go:356] "Generic (PLEG): container finished" podID="bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" containerID="2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7" exitCode=0 Sep 17 02:13:24.382275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.381897 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" Sep 17 02:13:24.382275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.381899 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" event={"ID":"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7","Type":"ContainerDied","Data":"2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7"} Sep 17 02:13:24.382275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.381952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd" event={"ID":"bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7","Type":"ContainerDied","Data":"b1b8a8feb447fce51e066486c7bf6ffdb72927aa440810664ec556f0a04475b6"} Sep 17 02:13:24.382275 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.381975 2582 scope.go:117] "RemoveContainer" containerID="2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7" Sep 17 02:13:24.390753 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.390738 2582 scope.go:117] "RemoveContainer" containerID="2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7" Sep 17 02:13:24.391030 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:13:24.391010 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7\": container with ID starting with 2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7 not found: ID does not exist" containerID="2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7" Sep 17 02:13:24.391095 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.391041 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7"} err="failed to get container status \"2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7\": rpc error: code = NotFound desc = could not find container \"2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7\": container with ID starting with 2a1dcb178959435621e8e59129d08af3a6fa8e025a4824fbd7409c40440e4dc7 not found: ID does not exist" Sep 17 02:13:24.402049 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.402025 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd"] Sep 17 02:13:24.407924 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.407905 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f03337-5f55cf5878-n7ltd"] Sep 17 02:13:24.874044 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:24.874009 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:13:25.679270 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:25.679235 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" path="/var/lib/kubelet/pods/bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7/volumes" Sep 17 02:13:29.669267 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:29.669173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:29.769964 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:29.769934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:29.971633 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:29.971601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:13:33.872290 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:33.872252 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:13:36.072919 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:36.072888 2582 scope.go:117] "RemoveContainer" containerID="08b41eb384e16071a1488ff7e872b1d857768c118c55cb4ae338fcfd8250cdb5" Sep 17 02:13:36.081046 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:13:36.081025 2582 scope.go:117] "RemoveContainer" containerID="c7033da96860bbae2fbd4c6e61330db54bdfee51d6b3d471ad7224edac0af73c" Sep 17 02:14:36.113479 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:14:36.113447 2582 scope.go:117] "RemoveContainer" containerID="ab250d18abde5209a1e157516ce5685530055856057b2d800035c1af79326ae0" Sep 17 02:14:36.122665 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:14:36.122639 2582 scope.go:117] "RemoveContainer" containerID="210f53f1a18697b63013e0012f06119188ce1f0a29afb85efd1a74e5c7150783" Sep 17 02:14:55.228403 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:14:55.228362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:14:55.965923 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:14:55.965888 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:14:56.811666 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:14:56.811635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:15:20.972991 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:20.972955 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:15:27.665382 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:27.665345 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:15:30.274920 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:30.274875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:15:33.668667 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:33.668629 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:15:41.767377 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:41.767343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:15:46.524733 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:46.524684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:15:46.625493 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:15:46.625459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:16:00.263497 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:00.263457 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:16:06.373226 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:06.373174 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:16:07.967185 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:07.967143 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd"] Sep 17 02:16:07.967653 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:07.967372 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" podUID="14decd50-2064-4ccc-ac01-1ddeaba37b4b" containerName="manager" containerID="cri-o://317bb5dbfcd1e33c8a1ec83c81f1c0f1928c945e6c2c27450c86edcd957d97c5" gracePeriod=10 Sep 17 02:16:09.191071 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.190979 2582 generic.go:356] "Generic (PLEG): container finished" podID="14decd50-2064-4ccc-ac01-1ddeaba37b4b" containerID="317bb5dbfcd1e33c8a1ec83c81f1c0f1928c945e6c2c27450c86edcd957d97c5" exitCode=0 Sep 17 02:16:09.191071 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.191054 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" event={"ID":"14decd50-2064-4ccc-ac01-1ddeaba37b4b","Type":"ContainerDied","Data":"317bb5dbfcd1e33c8a1ec83c81f1c0f1928c945e6c2c27450c86edcd957d97c5"} Sep 17 02:16:09.515077 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.515051 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 02:16:09.637443 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.637408 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v26s7\" (UniqueName: \"kubernetes.io/projected/14decd50-2064-4ccc-ac01-1ddeaba37b4b-kube-api-access-v26s7\") pod \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " Sep 17 02:16:09.637620 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.637492 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14decd50-2064-4ccc-ac01-1ddeaba37b4b-extensions-socket-volume\") pod \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\" (UID: \"14decd50-2064-4ccc-ac01-1ddeaba37b4b\") " Sep 17 02:16:09.637888 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.637853 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14decd50-2064-4ccc-ac01-1ddeaba37b4b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "14decd50-2064-4ccc-ac01-1ddeaba37b4b" (UID: "14decd50-2064-4ccc-ac01-1ddeaba37b4b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:16:09.639553 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.639530 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14decd50-2064-4ccc-ac01-1ddeaba37b4b-kube-api-access-v26s7" (OuterVolumeSpecName: "kube-api-access-v26s7") pod "14decd50-2064-4ccc-ac01-1ddeaba37b4b" (UID: "14decd50-2064-4ccc-ac01-1ddeaba37b4b"). InnerVolumeSpecName "kube-api-access-v26s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:16:09.738571 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.738537 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26s7\" (UniqueName: \"kubernetes.io/projected/14decd50-2064-4ccc-ac01-1ddeaba37b4b-kube-api-access-v26s7\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:16:09.738571 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:09.738565 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14decd50-2064-4ccc-ac01-1ddeaba37b4b-extensions-socket-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:16:10.197099 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:10.197070 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" Sep 17 02:16:10.197522 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:10.197114 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd" event={"ID":"14decd50-2064-4ccc-ac01-1ddeaba37b4b","Type":"ContainerDied","Data":"3d5718676b8c1a8469af6beaad4619a9d8e6ba6e206db5e42391c200dc7e5e0c"} Sep 17 02:16:10.197522 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:10.197178 2582 scope.go:117] "RemoveContainer" containerID="317bb5dbfcd1e33c8a1ec83c81f1c0f1928c945e6c2c27450c86edcd957d97c5" Sep 17 02:16:10.213737 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:10.213715 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd"] Sep 17 02:16:10.216674 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:10.216652 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6mnsd"] Sep 17 02:16:11.679112 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:16:11.679079 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14decd50-2064-4ccc-ac01-1ddeaba37b4b" path="/var/lib/kubelet/pods/14decd50-2064-4ccc-ac01-1ddeaba37b4b/volumes" Sep 17 02:17:08.799784 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.799744 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f"] Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800084 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bc4649-6e03-4967-8222-461ccabe913a" containerName="revoke-keys" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800095 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bc4649-6e03-4967-8222-461ccabe913a" containerName="revoke-keys" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800109 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14decd50-2064-4ccc-ac01-1ddeaba37b4b" containerName="manager" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800114 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="14decd50-2064-4ccc-ac01-1ddeaba37b4b" containerName="manager" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800128 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" containerName="payload-pre-processing" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800134 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" containerName="payload-pre-processing" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800196 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bc4649-6e03-4967-8222-461ccabe913a" containerName="revoke-keys" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800204 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="14decd50-2064-4ccc-ac01-1ddeaba37b4b" containerName="manager" Sep 17 02:17:08.800373 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.800224 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7a724e-4ea9-43b9-87c0-bcc8433ef4e7" containerName="payload-pre-processing" Sep 17 02:17:08.803195 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.803177 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:08.805688 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.805663 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n2jw2\"" Sep 17 02:17:08.813356 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.813321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f"] Sep 17 02:17:08.915366 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.915321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwn4s\" (UniqueName: \"kubernetes.io/projected/043b6b19-d900-4534-89a1-8727c25ab60e-kube-api-access-wwn4s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kch9f\" (UID: \"043b6b19-d900-4534-89a1-8727c25ab60e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:08.915559 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:08.915464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/043b6b19-d900-4534-89a1-8727c25ab60e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kch9f\" (UID: \"043b6b19-d900-4534-89a1-8727c25ab60e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:09.016105 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.016059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/043b6b19-d900-4534-89a1-8727c25ab60e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kch9f\" (UID: \"043b6b19-d900-4534-89a1-8727c25ab60e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:09.016351 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.016151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwn4s\" (UniqueName: \"kubernetes.io/projected/043b6b19-d900-4534-89a1-8727c25ab60e-kube-api-access-wwn4s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kch9f\" (UID: \"043b6b19-d900-4534-89a1-8727c25ab60e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:09.016503 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.016478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/043b6b19-d900-4534-89a1-8727c25ab60e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kch9f\" (UID: \"043b6b19-d900-4534-89a1-8727c25ab60e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:09.024645 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.024619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwn4s\" (UniqueName: \"kubernetes.io/projected/043b6b19-d900-4534-89a1-8727c25ab60e-kube-api-access-wwn4s\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kch9f\" (UID: \"043b6b19-d900-4534-89a1-8727c25ab60e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:09.115379 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.115282 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:09.406422 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.406393 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f"] Sep 17 02:17:09.408915 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:17:09.408870 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043b6b19_d900_4534_89a1_8727c25ab60e.slice/crio-d986882fd8f8af3d1b850fe7ca82aa2000e469611d133e4c067ad8c9d54ab45b WatchSource:0}: Error finding container d986882fd8f8af3d1b850fe7ca82aa2000e469611d133e4c067ad8c9d54ab45b: Status 404 returned error can't find the container with id d986882fd8f8af3d1b850fe7ca82aa2000e469611d133e4c067ad8c9d54ab45b Sep 17 02:17:09.483905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:09.483873 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" event={"ID":"043b6b19-d900-4534-89a1-8727c25ab60e","Type":"ContainerStarted","Data":"d986882fd8f8af3d1b850fe7ca82aa2000e469611d133e4c067ad8c9d54ab45b"} Sep 17 02:17:10.491464 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:10.491429 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" event={"ID":"043b6b19-d900-4534-89a1-8727c25ab60e","Type":"ContainerStarted","Data":"cd6f29904d7d3089ebd4a2db0a6f18c25a9d17898f99457f79c6c8b54ed81d0c"} Sep 17 02:17:10.491866 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:10.491493 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:10.518357 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:10.518305 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" podStartSLOduration=2.518288929 podStartE2EDuration="2.518288929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:17:10.512731294 +0000 UTC m=+2375.347783627" watchObservedRunningTime="2026-09-17 02:17:10.518288929 +0000 UTC m=+2375.353341254" Sep 17 02:17:21.499506 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:21.499468 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kch9f" Sep 17 02:17:33.666962 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:33.666917 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:17:36.172895 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:36.172865 2582 scope.go:117] "RemoveContainer" containerID="8de0ccbde69ecb28d91b74965d7f5daf78f6b71f2dd5a05e8dd085fec60f1033" Sep 17 02:17:36.181052 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:36.181032 2582 scope.go:117] "RemoveContainer" containerID="0ae225f7417cb87e5c8cf4d0066d3f8c0c1600c56232a827c92dd3e4778f8a46" Sep 17 02:17:41.364861 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:41.364830 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:17:45.075358 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.075327 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c"] Sep 17 02:17:45.078567 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.078545 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.080832 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.080809 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-65334296-kserve-self-signed-certs\"" Sep 17 02:17:45.088093 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.088067 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c"] Sep 17 02:17:45.120669 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.120642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:17:45.206604 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.206573 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tmp-dir\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.206712 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.206638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tls-certs\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.206782 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.206709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-dshm\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.206782 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.206751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8bg\" (UniqueName: \"kubernetes.io/projected/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-kube-api-access-nt8bg\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.206896 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.206826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-model-cache\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.206896 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.206858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-home\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307435 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-model-cache\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-home\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tmp-dir\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307532 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307515 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tls-certs\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307696 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-dshm\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307696 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8bg\" (UniqueName: \"kubernetes.io/projected/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-kube-api-access-nt8bg\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307831 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-model-cache\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307894 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307837 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-home\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.307936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.307893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tmp-dir\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.309747 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.309727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-dshm\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.309849 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.309832 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tls-certs\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.314919 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.314902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8bg\" (UniqueName: \"kubernetes.io/projected/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-kube-api-access-nt8bg\") pod \"e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.388460 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.388405 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:45.655614 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:45.655587 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c"] Sep 17 02:17:45.657645 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:17:45.657616 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd6140a_a37e_4e1a_b529_94bfa9bbde9a.slice/crio-a1ce16ef34f6dd06de22bcadb34f2d696e95adf85caa5b900d4aef6fec5108d6 WatchSource:0}: Error finding container a1ce16ef34f6dd06de22bcadb34f2d696e95adf85caa5b900d4aef6fec5108d6: Status 404 returned error can't find the container with id a1ce16ef34f6dd06de22bcadb34f2d696e95adf85caa5b900d4aef6fec5108d6 Sep 17 02:17:46.667778 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:46.667741 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" event={"ID":"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a","Type":"ContainerStarted","Data":"27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813"} Sep 17 02:17:46.667778 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:46.667784 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" event={"ID":"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a","Type":"ContainerStarted","Data":"a1ce16ef34f6dd06de22bcadb34f2d696e95adf85caa5b900d4aef6fec5108d6"} Sep 17 02:17:46.683985 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:46.683935 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" podStartSLOduration=1.683919447 podStartE2EDuration="1.683919447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:17:46.682127385 +0000 UTC m=+2411.517179720" watchObservedRunningTime="2026-09-17 02:17:46.683919447 +0000 UTC m=+2411.518971780" Sep 17 02:17:47.368756 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:47.368721 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:17:55.388857 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:55.388817 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:55.389417 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:55.388867 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:55.401490 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:55.401458 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:55.729063 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:55.729039 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:17:59.486243 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:17:59.486128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:18:02.577055 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:02.574007 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:18:03.585490 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:03.585452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:18:20.379485 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:20.379449 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c"] Sep 17 02:18:20.397759 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:20.397729 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:18:21.909043 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:21.909008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:18:22.574702 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:22.574661 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:18:25.965723 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:25.965677 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:18:31.271930 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:31.271894 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:18:35.399001 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.398965 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" podUID="7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" containerName="main" containerID="cri-o://27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813" gracePeriod=45 Sep 17 02:18:35.643224 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.643187 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:18:35.715797 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.715770 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt8bg\" (UniqueName: \"kubernetes.io/projected/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-kube-api-access-nt8bg\") pod \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " Sep 17 02:18:35.715936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.715812 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tmp-dir\") pod \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " Sep 17 02:18:35.715936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.715850 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-dshm\") pod \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " Sep 17 02:18:35.715936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.715871 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-home\") pod \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " Sep 17 02:18:35.715936 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.715901 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tls-certs\") pod \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " Sep 17 02:18:35.716157 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.715961 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-model-cache\") pod \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\" (UID: \"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a\") " Sep 17 02:18:35.716237 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.716160 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" (UID: "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:18:35.716237 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.716167 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-home" (OuterVolumeSpecName: "home") pod "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" (UID: "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:18:35.716413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.716313 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tmp-dir\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:18:35.716413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.716333 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-home\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:18:35.716413 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.716369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" (UID: "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:18:35.718016 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.717989 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-dshm" (OuterVolumeSpecName: "dshm") pod "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" (UID: "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:18:35.718121 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.717989 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-kube-api-access-nt8bg" (OuterVolumeSpecName: "kube-api-access-nt8bg") pod "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" (UID: "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a"). InnerVolumeSpecName "kube-api-access-nt8bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:18:35.718171 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.718151 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" (UID: "7bd6140a-a37e-4e1a-b529-94bfa9bbde9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:18:35.817081 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.817060 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:18:35.817081 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.817079 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8bg\" (UniqueName: \"kubernetes.io/projected/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-kube-api-access-nt8bg\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:18:35.817205 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.817089 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-dshm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:18:35.817205 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.817097 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a-tls-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\"" Sep 17 02:18:35.906848 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.906820 2582 generic.go:356] "Generic (PLEG): container finished" podID="7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" containerID="27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813" exitCode=0 Sep 17 02:18:35.906942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.906857 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" Sep 17 02:18:35.906942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.906878 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" event={"ID":"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a","Type":"ContainerDied","Data":"27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813"} Sep 17 02:18:35.906942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.906908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c" event={"ID":"7bd6140a-a37e-4e1a-b529-94bfa9bbde9a","Type":"ContainerDied","Data":"a1ce16ef34f6dd06de22bcadb34f2d696e95adf85caa5b900d4aef6fec5108d6"} Sep 17 02:18:35.906942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.906928 2582 scope.go:117] "RemoveContainer" containerID="27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813" Sep 17 02:18:35.917041 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.916738 2582 scope.go:117] "RemoveContainer" containerID="27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813" Sep 17 02:18:35.917969 ip-10-0-142-18 kubenswrapper[2582]: E0917 02:18:35.917938 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813\": container with ID starting with 27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813 not found: ID does not exist" containerID="27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813" Sep 17 02:18:35.918070 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.917980 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813"} err="failed to get container status \"27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813\": rpc error: code = NotFound desc = could not find container \"27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813\": container with ID starting with 27ed2a4d7928c861187fb3ee6b1f38db7ee587ee593275df78fe6d65422bd813 not found: ID does not exist" Sep 17 02:18:35.930698 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.930674 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c"] Sep 17 02:18:35.934048 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:35.934027 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-65334296-kserve-7d6dbf9c99-txq9c"] Sep 17 02:18:37.679549 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:37.679513 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" path="/var/lib/kubelet/pods/7bd6140a-a37e-4e1a-b529-94bfa9bbde9a/volumes" Sep 17 02:18:39.173899 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:39.173867 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:18:45.174112 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:45.174078 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:18:50.368419 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:50.368388 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:18:55.968659 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:18:55.968623 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:19:02.667818 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:02.667780 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wjsqq"] Sep 17 02:19:06.255124 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.255081 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nlq6t/must-gather-7nwcv"] Sep 17 02:19:06.255606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.255494 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" containerName="main" Sep 17 02:19:06.255606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.255508 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" containerName="main" Sep 17 02:19:06.255606 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.255596 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd6140a-a37e-4e1a-b529-94bfa9bbde9a" containerName="main" Sep 17 02:19:06.258897 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.258876 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.261073 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.261050 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nlq6t\"/\"default-dockercfg-cl7v2\"" Sep 17 02:19:06.261197 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.261050 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nlq6t\"/\"openshift-service-ca.crt\"" Sep 17 02:19:06.261197 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.261162 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nlq6t\"/\"kube-root-ca.crt\"" Sep 17 02:19:06.268792 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.268770 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nlq6t/must-gather-7nwcv"] Sep 17 02:19:06.278445 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.278420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:19:06.339593 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.339570 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:19:06.348341 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.348311 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkp4r\" (UniqueName: \"kubernetes.io/projected/7bb75d06-166c-4743-97be-de664a321dc3-kube-api-access-bkp4r\") pod \"must-gather-7nwcv\" (UID: \"7bb75d06-166c-4743-97be-de664a321dc3\") " pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.348417 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.348359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bb75d06-166c-4743-97be-de664a321dc3-must-gather-output\") pod \"must-gather-7nwcv\" (UID: \"7bb75d06-166c-4743-97be-de664a321dc3\") " pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.448942 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.448918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkp4r\" (UniqueName: \"kubernetes.io/projected/7bb75d06-166c-4743-97be-de664a321dc3-kube-api-access-bkp4r\") pod \"must-gather-7nwcv\" (UID: \"7bb75d06-166c-4743-97be-de664a321dc3\") " pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.449042 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.448956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bb75d06-166c-4743-97be-de664a321dc3-must-gather-output\") pod \"must-gather-7nwcv\" (UID: \"7bb75d06-166c-4743-97be-de664a321dc3\") " pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.449261 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.449246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bb75d06-166c-4743-97be-de664a321dc3-must-gather-output\") pod \"must-gather-7nwcv\" (UID: \"7bb75d06-166c-4743-97be-de664a321dc3\") " pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.459473 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.459450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkp4r\" (UniqueName: \"kubernetes.io/projected/7bb75d06-166c-4743-97be-de664a321dc3-kube-api-access-bkp4r\") pod \"must-gather-7nwcv\" (UID: \"7bb75d06-166c-4743-97be-de664a321dc3\") " pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.568949 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.568885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nlq6t/must-gather-7nwcv" Sep 17 02:19:06.839523 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.839494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nlq6t/must-gather-7nwcv"] Sep 17 02:19:06.841494 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:19:06.841464 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bb75d06_166c_4743_97be_de664a321dc3.slice/crio-dc18d238a9fdb0749e01b839dad4eb9b5c579c8025d171e4230c930e870071ce WatchSource:0}: Error finding container dc18d238a9fdb0749e01b839dad4eb9b5c579c8025d171e4230c930e870071ce: Status 404 returned error can't find the container with id dc18d238a9fdb0749e01b839dad4eb9b5c579c8025d171e4230c930e870071ce Sep 17 02:19:06.843559 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:06.843534 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 02:19:07.054698 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:07.054665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/must-gather-7nwcv" event={"ID":"7bb75d06-166c-4743-97be-de664a321dc3","Type":"ContainerStarted","Data":"dc18d238a9fdb0749e01b839dad4eb9b5c579c8025d171e4230c930e870071ce"} Sep 17 02:19:08.063010 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:08.062954 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/must-gather-7nwcv" event={"ID":"7bb75d06-166c-4743-97be-de664a321dc3","Type":"ContainerStarted","Data":"439fdc77d282bf9c290e4d133bb3f66c5f71eac908a8930524eb210dc1192bc6"} Sep 17 02:19:08.063010 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:08.063009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/must-gather-7nwcv" event={"ID":"7bb75d06-166c-4743-97be-de664a321dc3","Type":"ContainerStarted","Data":"66d1c1e5b2d49455ebde8c40a322c841d9087c9c80f39967ab7a0f2666a93c6e"} Sep 17 02:19:08.077079 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:08.077034 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nlq6t/must-gather-7nwcv" podStartSLOduration=1.317780445 podStartE2EDuration="2.077018473s" totalImagesPullingTime="759.238028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:19:08.076504069 +0000 UTC m=+2492.911556401" watchObservedRunningTime="2026-09-17 02:19:08.077018473 +0000 UTC m=+2492.912070843" Sep 17 02:19:09.217765 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:09.217686 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kjf7c_cbb95db5-3662-41bf-873e-e2e2e34947fc/global-pull-secret-syncer/0.log" Sep 17 02:19:09.275164 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:09.275129 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" Sep 17 02:19:09.354288 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:09.354254 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-18.ec2.internal_08401cdaa04694297668145e5dc9fe19/haproxy/0.log" Sep 17 02:19:14.197784 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:14.197701 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-kch9f_043b6b19-d900-4534-89a1-8727c25ab60e/manager/0.log" Sep 17 02:19:14.229234 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:14.229183 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wjsqq_100eb694-9a72-4066-a9d4-2b4dce3e430c/limitador/0.log" Sep 17 02:19:16.095424 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.095389 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxnxg_fc7568ca-af7a-4d75-842b-dd358c7b1f3d/node-exporter/0.log" Sep 17 02:19:16.115101 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.115056 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxnxg_fc7568ca-af7a-4d75-842b-dd358c7b1f3d/kube-rbac-proxy/0.log" Sep 17 02:19:16.136358 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.136326 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxnxg_fc7568ca-af7a-4d75-842b-dd358c7b1f3d/init-textfile/0.log" Sep 17 02:19:16.268051 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.268016 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/prometheus/0.log" Sep 17 02:19:16.287397 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.287366 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/config-reloader/0.log" Sep 17 02:19:16.309596 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.309555 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/thanos-sidecar/0.log" Sep 17 02:19:16.334780 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.334751 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/kube-rbac-proxy-web/0.log" Sep 17 02:19:16.357918 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.357898 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/kube-rbac-proxy/0.log" Sep 17 02:19:16.381730 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.381707 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/kube-rbac-proxy-thanos/0.log" Sep 17 02:19:16.405230 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.405179 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_71d285eb-613e-49bd-9559-d7fce5e94b03/init-config-reloader/0.log" Sep 17 02:19:16.447972 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.447943 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5vbf8_2d4f62d6-50d2-4dec-a191-da64d94c1362/prometheus-operator/0.log" Sep 17 02:19:16.466576 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.466531 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5vbf8_2d4f62d6-50d2-4dec-a191-da64d94c1362/kube-rbac-proxy/0.log" Sep 17 02:19:16.489514 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.489484 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-5jtkq_8625ef2c-2727-4c73-94d5-de6ef4000ec8/prometheus-operator-admission-webhook/0.log" Sep 17 02:19:16.603958 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.603692 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77769479f9-4b4bk_cfd268f3-cac2-4cdd-add7-17120729f3fb/thanos-query/0.log" Sep 17 02:19:16.640905 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.640881 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77769479f9-4b4bk_cfd268f3-cac2-4cdd-add7-17120729f3fb/kube-rbac-proxy-web/0.log" Sep 17 02:19:16.674769 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.674741 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77769479f9-4b4bk_cfd268f3-cac2-4cdd-add7-17120729f3fb/kube-rbac-proxy/0.log" Sep 17 02:19:16.697433 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.697405 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77769479f9-4b4bk_cfd268f3-cac2-4cdd-add7-17120729f3fb/prom-label-proxy/0.log" Sep 17 02:19:16.718350 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.718326 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77769479f9-4b4bk_cfd268f3-cac2-4cdd-add7-17120729f3fb/kube-rbac-proxy-rules/0.log" Sep 17 02:19:16.765781 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:16.765745 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77769479f9-4b4bk_cfd268f3-cac2-4cdd-add7-17120729f3fb/kube-rbac-proxy-metrics/0.log" Sep 17 02:19:19.809461 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.809427 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk"] Sep 17 02:19:19.813255 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.813230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.821634 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.821610 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk"] Sep 17 02:19:19.890875 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.890691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbzs\" (UniqueName: \"kubernetes.io/projected/a2b68909-ed23-4cc3-8170-8112776e09bd-kube-api-access-tjbzs\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.890875 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.890757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-podres\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.890875 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.890789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-lib-modules\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.891187 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.890899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-proc\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.891187 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.890962 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-sys\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992306 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-proc\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-sys\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbzs\" (UniqueName: \"kubernetes.io/projected/a2b68909-ed23-4cc3-8170-8112776e09bd-kube-api-access-tjbzs\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992398 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-proc\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-podres\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-sys\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992486 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-lib-modules\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992815 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-podres\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:19.992815 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:19.992660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2b68909-ed23-4cc3-8170-8112776e09bd-lib-modules\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:20.000637 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:20.000590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbzs\" (UniqueName: \"kubernetes.io/projected/a2b68909-ed23-4cc3-8170-8112776e09bd-kube-api-access-tjbzs\") pod \"perf-node-gather-daemonset-ft7sk\" (UID: \"a2b68909-ed23-4cc3-8170-8112776e09bd\") " pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:20.127441 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:20.127103 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:20.422922 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:20.422899 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk"] Sep 17 02:19:20.425895 ip-10-0-142-18 kubenswrapper[2582]: W0917 02:19:20.425865 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2b68909_ed23_4cc3_8170_8112776e09bd.slice/crio-9b50aa2a5b3e7be485b8d3c6a69e497ba7d1e65c151a2e97caadbc28b807e681 WatchSource:0}: Error finding container 9b50aa2a5b3e7be485b8d3c6a69e497ba7d1e65c151a2e97caadbc28b807e681: Status 404 returned error can't find the container with id 9b50aa2a5b3e7be485b8d3c6a69e497ba7d1e65c151a2e97caadbc28b807e681 Sep 17 02:19:21.158598 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.158563 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" event={"ID":"a2b68909-ed23-4cc3-8170-8112776e09bd","Type":"ContainerStarted","Data":"6a08cd58c1af55e6c1271d855fccf906b24461379342aae184da3e6bbcc7adc6"} Sep 17 02:19:21.159023 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.158661 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:21.159023 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.158688 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" event={"ID":"a2b68909-ed23-4cc3-8170-8112776e09bd","Type":"ContainerStarted","Data":"9b50aa2a5b3e7be485b8d3c6a69e497ba7d1e65c151a2e97caadbc28b807e681"} Sep 17 02:19:21.174608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.174563 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" podStartSLOduration=2.1745478990000002 podStartE2EDuration="2.174547899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 02:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:19:21.172856085 +0000 UTC m=+2506.007908434" watchObservedRunningTime="2026-09-17 02:19:21.174547899 +0000 UTC m=+2506.009600265" Sep 17 02:19:21.353200 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.353166 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q2l8r_cdcd7a2a-0e54-4fef-abb6-a1046f64fa21/dns/0.log" Sep 17 02:19:21.372910 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.372881 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q2l8r_cdcd7a2a-0e54-4fef-abb6-a1046f64fa21/kube-rbac-proxy/0.log" Sep 17 02:19:21.393053 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.393032 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-498hn_5c521d6a-4e1e-4e0f-8e8f-21289c64a2a0/dns-node-resolver/0.log" Sep 17 02:19:21.955873 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:21.955844 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4687l_7c84a533-aeac-4d80-b820-f72b570c337d/node-ca/0.log" Sep 17 02:19:23.286802 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:23.286772 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79c4dfc8-9sg2x_653dc08d-064b-469c-8583-b87821c2d482/kube-auth-proxy/0.log" Sep 17 02:19:24.260091 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:24.260057 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sd79z_68b2a83b-9d41-4051-a06c-797d0e57edd2/serve-healthcheck-canary/0.log" Sep 17 02:19:24.930019 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:24.929978 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hn297_f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc/kube-rbac-proxy/0.log" Sep 17 02:19:24.950004 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:24.949977 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hn297_f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc/exporter/0.log" Sep 17 02:19:24.969743 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:24.969719 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hn297_f108a8c3-f01c-4c0a-ae6e-c9336ce4ffbc/extractor/0.log" Sep 17 02:19:26.568015 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:26.567965 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:19:27.174000 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:27.173969 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nlq6t/perf-node-gather-daemonset-ft7sk" Sep 17 02:19:27.739404 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:27.739369 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f459ff6-d8jlr_14d1372a-5a9e-4db3-8768-5ad476bf3476/manager/0.log" Sep 17 02:19:27.758637 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:27.758615 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5f459ff6-d8jlr_14d1372a-5a9e-4db3-8768-5ad476bf3476/copy-manifests/0.log" Sep 17 02:19:29.121225 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:29.121130 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c846775dd-cdsbl_498ead04-90ec-45ba-bafb-aebafb2fda1c/manager/0.log" Sep 17 02:19:29.914330 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:29.914295 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79c4dfc8-9sg2x_653dc08d-064b-469c-8583-b87821c2d482/kube-auth-proxy/0.log" Sep 17 02:19:31.815924 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:31.815816 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b79c4dfc8-9sg2x_653dc08d-064b-469c-8583-b87821c2d482/kube-auth-proxy/0.log" Sep 17 02:19:32.744481 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:32.744441 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6rdjf_502a29ca-ecd3-481a-b86d-f66314d15ae1/konnectivity-agent/0.log" limit=1024 Sep 17 02:19:34.793719 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:34.793679 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-7hm4g_5ad2fb6c-1185-4211-8e82-ae810c248e8b/kube-storage-version-migrator-operator/1.log" Sep 17 02:19:34.795354 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:34.795328 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-7hm4g_5ad2fb6c-1185-4211-8e82-ae810c248e8b/kube-storage-version-migrator-operator/0.log" Sep 17 02:19:35.735164 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:35.735129 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-66gs2_26831bc7-27ea-414d-9685-58b9e1d017d8/kube-multus/0.log" Sep 17 02:19:36.071767 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.071687 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/kube-multus-additional-cni-plugins/0.log" Sep 17 02:19:36.089294 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.089259 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/egress-router-binary-copy/0.log" Sep 17 02:19:36.107416 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.107396 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/cni-plugins/0.log" Sep 17 02:19:36.125999 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.125976 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/bond-cni-plugin/0.log" Sep 17 02:19:36.145329 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.145305 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/routeoverride-cni/0.log" Sep 17 02:19:36.176931 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.176915 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/whereabouts-cni-bincopy/0.log" Sep 17 02:19:36.195036 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.195011 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhrrm_bde4c47b-878a-400d-a28b-4eb30e925a9f/whereabouts-cni/0.log" Sep 17 02:19:36.214794 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.214773 2582 scope.go:117] "RemoveContainer" containerID="aba627e764fa5836f66dcb53b1e8b5a4e1d671c39b4171e872c06ed9991e45d3" Sep 17 02:19:36.226677 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.226659 2582 scope.go:117] "RemoveContainer" containerID="ed430dc6abce6244cc0f1bd58118addef3ef393fcb4c55bdedc66fdf31ad3425" Sep 17 02:19:36.235948 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.235927 2582 scope.go:117] "RemoveContainer" containerID="cff5ff27dc368db8eaecc3595d724f7bd0e6d6e4cc2e8a6464aefc226312d417" Sep 17 02:19:36.349608 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.349523 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vkjcl_bf91a16c-6494-4e30-a6e2-1cdefa9856c6/network-metrics-daemon/0.log" Sep 17 02:19:36.362832 ip-10-0-142-18 kubenswrapper[2582]: I0917 02:19:36.362811 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vkjcl_bf91a16c-6494-4e30-a6e2-1cdefa9856c6/kube-rbac-proxy/0.log"