Sep 08 03:58:25.880182 ip-10-0-130-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 03:58:25.880193 ip-10-0-130-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 03:58:25.880203 ip-10-0-130-111 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 03:58:25.880479 ip-10-0-130-111 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 03:58:36.129124 ip-10-0-130-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 03:58:36.129142 ip-10-0-130-111 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 947a30d063ef414e8ce22ec9ad49cff8 --
Sep 08 04:00:19.533132 ip-10-0-130-111 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 04:00:19.966967 ip-10-0-130-111 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 04:00:19.966967 ip-10-0-130-111 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 04:00:19.966967 ip-10-0-130-111 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 04:00:19.966967 ip-10-0-130-111 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 04:00:19.966967 ip-10-0-130-111 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 04:00:19.967766 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.967065    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 04:00:19.970543 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970521    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 04:00:19.970543 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970539    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 04:00:19.970543 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970544    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 04:00:19.970543 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970550    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970555    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970559    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970563    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970569    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970574    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970578    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970584    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970589    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970592    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970597    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970601    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970604    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970608    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970612    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970616    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970620    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970624    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970628    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 04:00:19.970822 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970632    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970636    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970640    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970644    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970655    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970661    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970666    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970670    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970674    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970678    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970682    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970686    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970689    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970693    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970698    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970703    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970708    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970712    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970716    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 04:00:19.971591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970721    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970728    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970733    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970751    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970756    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970761    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970765    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970769    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970773    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970777    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970781    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970785    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970789    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970793    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970797    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970802    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970806    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970810    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970814    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 04:00:19.972407 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970818    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970822    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970826    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970830    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970834    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970838    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970842    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970847    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970852    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970858    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970862    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970866    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970871    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970875    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970879    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970883    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970887    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970897    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970904    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970908    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 04:00:19.973163 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970912    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970917    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970921    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970925    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970929    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970934    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970938    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970942    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970947    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970951    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970955    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970959    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970964    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970968    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970972    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970976    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970981    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970985    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.970990    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972143    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 04:00:19.973644 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972154    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972159    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972163    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972167    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972173    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972178    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972182    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972186    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972190    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972195    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972204    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972209    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972216    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972222    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972228    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972232    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972237    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972241    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972245    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972249    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 04:00:19.974470 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972254    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972258    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972262    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972266    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972270    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972274    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972279    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972283    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972288    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972296    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972301    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972305    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972309    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972314    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972318    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972322    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972326    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972330    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972337    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 04:00:19.975474 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972342    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972349    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972354    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972361    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972367    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972372    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972376    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972381    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972385    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972390    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972394    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972398    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972402    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972406    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972409    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972414    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972417    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972421    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972425    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 04:00:19.976231 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972429    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972433    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972438    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972468    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972474    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972480    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972485    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972489    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972494    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972499    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972503    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972508    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972512    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972516    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972520    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972524    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972528    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972533    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972537    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972542    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 04:00:19.976683 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972546    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972550    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972554    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972558    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972562    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972569    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972573    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972577    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972582    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972586    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972590    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972594    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972598    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972603    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972607    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972611    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972616    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972620    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.972625    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 04:00:19.977299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.973973    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.973984    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.973990    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.973995    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.973999    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974004    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974009    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974013    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974017    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974021    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974026    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974030    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974035    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974039    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974043    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974047    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974052    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974056    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974060    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974065    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 04:00:19.977821 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974069    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974073    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974077    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974082    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974086    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974090    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974094    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974098    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974103    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974107    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974112    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974116    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974124    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974130    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974134    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974138    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974143    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974147    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974152    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 04:00:19.978299 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974157    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974161    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974165    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974169    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974174    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974178    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974182    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974186    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974190    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974194    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974199    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974203    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974207    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974211    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974216    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974220    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974224    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974228    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974232    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974237    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 04:00:19.979037 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974240    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974244    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974248    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974252    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974259    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974265    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974269    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974273    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974277    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974281    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974285    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974293    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974299    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974303    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974307    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974312    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974316    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974320    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974325    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 04:00:19.979531 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974329    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974333    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974337    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974341    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974346    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974350    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974354    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974358    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974362    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974367    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974371    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974376    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974380    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974384    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974389    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974393    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974396    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974401    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974405    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974409    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 04:00:19.980032 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.974413    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974504    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974563    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974573    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974580    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974586    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974591    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974599    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974606    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974611    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974615    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974622    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974627    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974632    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974636    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974640    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974645    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974649    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974654    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974661    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974665    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974670    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974674    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974680    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 04:00:19.980607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974686    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974691    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974695    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974700    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974704    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974709    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974713    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974718    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974722    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974729    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974734    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974756    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974760    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974765    2583 flags.go:64] FLAG: --enable-server="true"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974769    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974775    2583 flags.go:64] FLAG: --event-burst="100"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974780    2583 flags.go:64] FLAG: --event-qps="50"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974784    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974789    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974794    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974802    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974810    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974815    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974820    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974825    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 04:00:19.981232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974830    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974834    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974839    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974844    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974849    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974854    2583 flags.go:64] FLAG: --feature-gates=""
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974860    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974864    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974869    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974874    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974880    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974885    2583 flags.go:64] FLAG: --help="false"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974890    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-130-111.ec2.internal"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974895    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974899    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974904    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974909    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974915    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974920    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974925    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974929    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974934    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974939    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974944    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 04:00:19.981869 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974948    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974953    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974957    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974963    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974970    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974974    2583 flags.go:64] FLAG: --lock-file=""
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974979    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974984    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974988    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.974997    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975002    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975006    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975011    2583 flags.go:64] FLAG: --logging-format="text"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975016    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975021    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975025    2583 flags.go:64] FLAG: --manifest-url=""
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975030    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975036    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975041    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975048    2583 flags.go:64] FLAG: --max-pods="110"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975053    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975058    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975062    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975067    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975072    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 04:00:19.982447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975078    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975083    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975095    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975100    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975105    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975110    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975115    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975123    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975128    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975133    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975138    2583 flags.go:64] FLAG: --port="10250"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975143    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975150    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0865339d79f46a9d6"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975155    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975160    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975165    2583 flags.go:64] FLAG: --register-node="true"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975170    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975175    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975180    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975185    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975189    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975195    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975200    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975205    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975209    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 04:00:19.983074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975213    2583 flags.go:64] FLAG: --runonce="false"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975218    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975223    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975228    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975232    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975237    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975242    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975248    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975253    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975257    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975262    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975267    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975271    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975277    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975281    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975286    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975294    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975299    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975303    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975312    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975318    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975323    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975328    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975333    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975337    2583 flags.go:64] FLAG: --v="2"
Sep 08 04:00:19.983806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975344    2583 flags.go:64] FLAG: --version="false"
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975350    2583 flags.go:64] FLAG: --vmodule=""
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975356    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.975361    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975540    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975549    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975554    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975559    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975565    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975569    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975575    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975579    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975584    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975588    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975595    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975599    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975604    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975608    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975613    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975617    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 04:00:19.984453 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975621    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975625    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975629    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975633    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975637    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975642    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975647    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975652    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975657    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975661    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975665    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975669    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975673    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975677    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975682    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975686    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975690    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975694    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975698    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975702    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 04:00:19.984971 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975706    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975711    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975715    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975719    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975723    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975727    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975733    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975752    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975757    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975762    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975766    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975769    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975774    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975778    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975784    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975789    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975794    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975799    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975805    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975810    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 04:00:19.985451 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975814    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975824    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975829    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975833    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975838    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975842    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975846    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975850    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975855    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975859    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975863    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975869    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975875    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975880    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975884    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975888    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975892    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975896    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975904    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 04:00:19.985961 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975910    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975915    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975920    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975924    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975928    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975932    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975937    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975941    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975945    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975949    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975953    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975959    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975963    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975967    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975972    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975977    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975981    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975985    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975990    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975994    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 04:00:19.986430 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.975998    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.976002    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.976006    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.976011    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.976777    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.984790    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.984806    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.984821    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.984827    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984883    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984889    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984892    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984895    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984898    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984901    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 04:00:19.986941 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984904    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984907    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984910    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984913    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984916    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984919    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984922    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984925    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984928    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984931    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984933    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984936    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984938    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984941    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984943    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984946    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984949    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984952    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984954    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984957    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 04:00:19.987308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984960    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984963    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984965    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984969    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984972    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984974    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984977    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984980    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984983    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984985    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984988    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984992    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984996    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.984999    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985002    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985005    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985008    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985010    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985013    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 04:00:19.987878 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985015    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985018    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985022    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985026    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985029    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985032    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985035    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985037    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985040    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985043    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985045    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985048    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985050    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985052    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985055    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985057    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985060    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985063    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985066    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 04:00:19.988364 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985069    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985072    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985074    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985077    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985079    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985082    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985085    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985087    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985090    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985092    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985095    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985098    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985101    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985104    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985107    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985110    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985113    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985116    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985118    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 04:00:19.988840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985120    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985123    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985126    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985128    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985131    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985133    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985136    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985138    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985141    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985143    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985146    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985148    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985151    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985154    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985156    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 04:00:19.989316 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985159    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.985164    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985293    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985298    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985302    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985305    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985308    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985310    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985313    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985316    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985318    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985322    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985326    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985329    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985332    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 04:00:19.989676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985335    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985338    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985340    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985343    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985345    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985348    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985350    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985353    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985355    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985358    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985360    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985362    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985365    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985368    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985370    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985373    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985376    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985378    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985381    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985383    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 04:00:19.990069 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985386    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985388    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985391    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985393    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985396    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985399    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985401    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985404    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985406    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985408    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985411    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985413    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985416    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985418    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985421    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985423    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985425    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985429    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985432    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 04:00:19.990547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985435    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985438    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985441    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985444    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985447    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985449    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985452    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985455    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985458    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985461    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985464    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985466    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985469    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985471    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985474    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985476    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985479    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985482    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985484    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 04:00:19.991109 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985486    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985489    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985491    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985494    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985496    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985499    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985501    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985504    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985506    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985509    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985512    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985515    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985518    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985521    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985524    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985527    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985531    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985533    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985536    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 04:00:19.991567 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985538    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985541    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985544    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985547    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985550    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985553    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985555    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985558    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:19.985561    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.985565    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.985668    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.988042    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.989096    2583 server.go:1019] "Starting client certificate rotation"
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.989192    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 04:00:19.992049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.989227    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 04:00:19.992814 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.992800    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 04:00:19.994426 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:19.994412    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 04:00:20.006849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.006832    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 08 04:00:20.009972 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.009956    2583 log.go:25] "Validated CRI v1 image API"
Sep 08 04:00:20.012917 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.012897    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 04:00:20.020570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.020551    2583 fs.go:135] Filesystem UUIDs: map[3fe541d3-63f6-40c1-9a3d-ff20c54d0de2:/dev/nvme0n1p3 62f20d99-2fe1-4708-9819-45760821f463:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 08 04:00:20.020643 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.020569    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 04:00:20.026909 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.026798    2583 manager.go:216] Machine: {Timestamp:2026-09-08 04:00:20.025210073 +0000 UTC m=+0.383689628 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104481 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22242ee7a324771713ebb949cfa5e4 SystemUUID:ec22242e-e7a3-2477-1713-ebb949cfa5e4 BootID:947a30d0-63ef-414e-8ce2-2ec9ad49cff8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:b2:74:df:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:b2:74:df:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:70:74:6d:fa:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 04:00:20.026909 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.026906    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 04:00:20.027024 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.026979    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 04:00:20.027707 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.027690    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 04:00:20.030443 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.030420    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 04:00:20.030582 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.030446    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 04:00:20.031410 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.031400    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 04:00:20.031442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.031412    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 04:00:20.031442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.031420    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 04:00:20.031442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.031436    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 04:00:20.033432 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.033423    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 04:00:20.038003 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.037992    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 04:00:20.038124 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038115    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 04:00:20.038164 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038145    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 04:00:20.038164 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038155    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 04:00:20.038223 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038168    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 04:00:20.038223 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038177    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 08 04:00:20.038223 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038186    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 04:00:20.038485 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038463    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 04:00:20.038485 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.038483    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 04:00:20.041465 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.041448    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 04:00:20.044417 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044403    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 04:00:20.044476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044444    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 04:00:20.044592 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044583    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 04:00:20.044624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044597    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 04:00:20.044624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044603    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 04:00:20.044624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044608    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 04:00:20.044624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044616    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 04:00:20.044624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.044621    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 04:00:20.045290 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.045279    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 04:00:20.045321 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.045292    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 04:00:20.045321 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.045300    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 04:00:20.045321 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.045308    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 04:00:20.045401 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.045329    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 04:00:20.045401 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.045341    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 04:00:20.046505 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.046490    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 04:00:20.046538 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.046508    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 04:00:20.050419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.050406    2583 server.go:1268] "Started kubelet"
Sep 08 04:00:20.050537 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.050514    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 04:00:20.050589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.050514    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 04:00:20.050589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.050578    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 04:00:20.051204 ip-10-0-130-111 systemd[1]: Started Kubernetes Kubelet.
Sep 08 04:00:20.052115 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.052096    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 04:00:20.057257 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.057238    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 04:00:20.057349 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.057281    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 04:00:20.058077 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.058019    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 04:00:20.058077 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.058029    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 04:00:20.058077 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.058027    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 04:00:20.060625 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.060607    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 04:00:20.060714 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.060630    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 04:00:20.060781 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.060710    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.061338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061315    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 04:00:20.061421 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061394    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 04:00:20.061478 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061428    2583 factory.go:55] Registering systemd factory
Sep 08 04:00:20.061478 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061343    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 04:00:20.061478 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061443    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 08 04:00:20.061626 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.060214    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-111.ec2.internal.18d33c85431b0795  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.ec2.internal,UID:ip-10-0-130-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.ec2.internal,},FirstTimestamp:2026-09-08 04:00:20.050380693 +0000 UTC m=+0.408860244,LastTimestamp:2026-09-08 04:00:20.050380693 +0000 UTC m=+0.408860244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.ec2.internal,}"
Sep 08 04:00:20.061626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061546    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 04:00:20.061626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061556    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 04:00:20.061769 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061722    2583 factory.go:153] Registering CRI-O factory
Sep 08 04:00:20.061835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061733    2583 factory.go:223] Registration of the crio container factory successfully
Sep 08 04:00:20.061890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061880    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 04:00:20.061950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061894    2583 factory.go:103] Registering Raw factory
Sep 08 04:00:20.061950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.061905    2583 manager.go:1196] Started watching for new ooms in manager
Sep 08 04:00:20.062217 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.062204    2583 manager.go:349] Starting recovery of all containers
Sep 08 04:00:20.069937 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.069911    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 04:00:20.070081 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.069940    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 04:00:20.073724 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.073708    2583 manager.go:354] Recovery completed
Sep 08 04:00:20.078456 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.078444    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.080607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.080587    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.080671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.080623    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.080671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.080637    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.081382 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.081287    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 04:00:20.081442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.081386    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 04:00:20.081442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.081287    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.081521 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.081402    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 04:00:20.083901 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.082885    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-111.ec2.internal.18d33c8544e83f3d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.ec2.internal,UID:ip-10-0-130-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.ec2.internal,},FirstTimestamp:2026-09-08 04:00:20.080607037 +0000 UTC m=+0.439086591,LastTimestamp:2026-09-08 04:00:20.080607037 +0000 UTC m=+0.439086591,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.ec2.internal,}"
Sep 08 04:00:20.083992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.083912    2583 policy_none.go:49] "None policy: Start"
Sep 08 04:00:20.083992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.083929    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 04:00:20.083992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.083941    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 04:00:20.084139 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.084005    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.084139 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.084032    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.084139 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.084050    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.086094 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.086081    2583 policy_none.go:47] "Start"
Sep 08 04:00:20.097718 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.097656    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-111.ec2.internal.18d33c8544e8974f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.ec2.internal,UID:ip-10-0-130-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-111.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.ec2.internal,},FirstTimestamp:2026-09-08 04:00:20.080629583 +0000 UTC m=+0.439109135,LastTimestamp:2026-09-08 04:00:20.080629583 +0000 UTC m=+0.439109135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.ec2.internal,}"
Sep 08 04:00:20.107334 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.107259    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-111.ec2.internal.18d33c8544e8cb55  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.ec2.internal,UID:ip-10-0-130-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-111.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.ec2.internal,},FirstTimestamp:2026-09-08 04:00:20.080642901 +0000 UTC m=+0.439122452,LastTimestamp:2026-09-08 04:00:20.080642901 +0000 UTC m=+0.439122452,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.ec2.internal,}"
Sep 08 04:00:20.108636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.108616    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nd57t"
Sep 08 04:00:20.109478 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.109462    2583 manager.go:337] "Starting Device Plugin manager"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.109513    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.109526    2583 server.go:87] "Starting device plugin registration server"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.109812    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.109824    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.109992    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.110063    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.110072    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.110627    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.110674    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.116591    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-111.ec2.internal.18d33c8544e83f3d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-111.ec2.internal.18d33c8544e83f3d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.ec2.internal,UID:ip-10-0-130-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.ec2.internal,},FirstTimestamp:2026-09-08 04:00:20.080607037 +0000 UTC m=+0.439086591,LastTimestamp:2026-09-08 04:00:20.084018626 +0000 UTC m=+0.442498177,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.ec2.internal,}"
Sep 08 04:00:20.136765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.118026    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nd57t"
Sep 08 04:00:20.152159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.152135    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 04:00:20.153237 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.153220    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 04:00:20.153311 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.153262    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 04:00:20.153311 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.153280    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 04:00:20.153417 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.153338    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 04:00:20.155997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.155977    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 04:00:20.210351 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.210332    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.211357 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.211342    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.211436 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.211371    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.211436 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.211385    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.211436 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.211405    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.225819 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.225762    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.225819 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.225781    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-111.ec2.internal\": node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.253898 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.253883    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"]
Sep 08 04:00:20.253989 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.253933    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.254788 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.254775    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.254880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.254805    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.254880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.254819    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.256727 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.256713    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.256835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.256822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.256892 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.256846    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.258236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.258219    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.258331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.258248    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.258331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.258219    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.258331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.258289    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.258331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.258259    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.258331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.258309    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.260091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.260075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.260170 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.260108    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:20.260851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.260835    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:20.260940 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.260865    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:20.260940 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.260880    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:20.273282 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.273263    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-111.ec2.internal\" not found" node="ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.276862 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.276846    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-111.ec2.internal\" not found" node="ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.279072 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.279057    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.362680 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.362657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb5647e1cfe7a0d9b5f7192d30c19c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.ec2.internal\" (UID: \"bb5647e1cfe7a0d9b5f7192d30c19c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.362780 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.362704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b957b5120b5ad80e4bc017af6c8f213-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal\" (UID: \"9b957b5120b5ad80e4bc017af6c8f213\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.362780 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.362722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b957b5120b5ad80e4bc017af6c8f213-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal\" (UID: \"9b957b5120b5ad80e4bc017af6c8f213\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.379142 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.379120    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.462985 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.462957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b957b5120b5ad80e4bc017af6c8f213-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal\" (UID: \"9b957b5120b5ad80e4bc017af6c8f213\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.462985 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.462985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b957b5120b5ad80e4bc017af6c8f213-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal\" (UID: \"9b957b5120b5ad80e4bc017af6c8f213\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.463156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.463003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb5647e1cfe7a0d9b5f7192d30c19c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.ec2.internal\" (UID: \"bb5647e1cfe7a0d9b5f7192d30c19c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.463156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.463054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb5647e1cfe7a0d9b5f7192d30c19c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.ec2.internal\" (UID: \"bb5647e1cfe7a0d9b5f7192d30c19c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.463156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.463073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b957b5120b5ad80e4bc017af6c8f213-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal\" (UID: \"9b957b5120b5ad80e4bc017af6c8f213\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.463156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.463079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b957b5120b5ad80e4bc017af6c8f213-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal\" (UID: \"9b957b5120b5ad80e4bc017af6c8f213\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.479345 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.479295    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.576001 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.575976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.579228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.579213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:20.580652 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.580310    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.680761 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.680729    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.781296 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.781275    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.881753 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.881709    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.983086 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:20.983056    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:20.989712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.989691    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 04:00:20.989888 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:20.989869    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 04:00:21.083905 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:21.083844    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:21.121218 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.121177    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 03:55:20 +0000 UTC" deadline="2028-05-15 19:00:35.358017072 +0000 UTC"
Sep 08 04:00:21.121218 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.121217    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14775h0m14.236806571s"
Sep 08 04:00:21.162952 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.162929    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 04:00:21.164040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.164023    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 04:00:21.164133 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.164054    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 04:00:21.164133 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.164067    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeHasSufficientPID"
Sep 08 04:00:21.184437 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:21.184407    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:21.241334 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:21.241306    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5647e1cfe7a0d9b5f7192d30c19c7d.slice/crio-f4a1221d7af452a8fa7b5a6bbceafe328a57434cba6122414f82e0404eb67164 WatchSource:0}: Error finding container f4a1221d7af452a8fa7b5a6bbceafe328a57434cba6122414f82e0404eb67164: Status 404 returned error can't find the container with id f4a1221d7af452a8fa7b5a6bbceafe328a57434cba6122414f82e0404eb67164
Sep 08 04:00:21.244498 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.244483    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 04:00:21.285000 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:21.284975    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:21.344716 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.344668    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 04:00:21.385793 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:21.385770    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-111.ec2.internal\" not found"
Sep 08 04:00:21.471110 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.471088    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 04:00:21.561971 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.561948    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:21.573954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.573935    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 04:00:21.575068 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.575055    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal"
Sep 08 04:00:21.588249 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.588232    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 04:00:21.646893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:21.646852    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 04:00:22.041779 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.041757    2583 apiserver.go:52] "Watching apiserver"
Sep 08 04:00:22.049499 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.049481    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 04:00:22.049889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.049868    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-jp22z","kube-system/global-pull-secret-syncer-lrg6q","kube-system/konnectivity-agent-btsdv","kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v","openshift-cluster-node-tuning-operator/tuned-j65w8","openshift-dns/node-resolver-rjjg2","openshift-image-registry/node-ca-xjmjj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal","openshift-multus/multus-9nwc7","openshift-multus/multus-additional-cni-plugins-gts66","openshift-multus/network-metrics-daemon-kj9xx","openshift-network-diagnostics/network-check-target-bw44h","openshift-network-operator/iptables-alerter-zkzhz"]
Sep 08 04:00:22.052458 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.052437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.055849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.055580    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.055849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.055635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 04:00:22.055849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.055798    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.055849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.055820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.056118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.055951    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t54rl\""
Sep 08 04:00:22.058102 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058082    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 04:00:22.058272 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058256    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 04:00:22.058386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058337    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 04:00:22.058502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058463    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgnxt\""
Sep 08 04:00:22.058502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058470    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.058718 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.058808 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.058727    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.060921 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.060881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.061046 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.061007    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.061157 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.061142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.061329 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.061314    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 04:00:22.061748 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.061723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vkdrf\""
Sep 08 04:00:22.063314 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.063270    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 04:00:22.063938 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.063363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mkglr\""
Sep 08 04:00:22.064041 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.064018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.066383 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.066211    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ksnjb\""
Sep 08 04:00:22.066383 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.066262    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.066558 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.066420    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.066619 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.066567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.069252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.069060    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 04:00:22.069252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.069167    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vsdrm\""
Sep 08 04:00:22.069436 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.069406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.069436 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.069419    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 04:00:22.070771 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.070745    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b957b5120b5ad80e4bc017af6c8f213.slice/crio-bf90897aae22704f8e0fa43c0622fbf2b96fcd4bae87764d02ea9b50a67738d0 WatchSource:0}: Error finding container bf90897aae22704f8e0fa43c0622fbf2b96fcd4bae87764d02ea9b50a67738d0: Status 404 returned error can't find the container with id bf90897aae22704f8e0fa43c0622fbf2b96fcd4bae87764d02ea9b50a67738d0
Sep 08 04:00:22.071635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.071607    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 04:00:22.071882 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.071861    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.071882 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.071879    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bxmpk\""
Sep 08 04:00:22.072026 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.071980    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.072083 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.072067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.072132 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.072116    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 04:00:22.073171 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.073171 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.073323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2988\" (UniqueName: \"kubernetes.io/projected/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-kube-api-access-z2988\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.073323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-multus-certs\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-etc-kubernetes\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-modprobe-d\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-kubernetes\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-netns\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-etc-selinux\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqb46\" (UniqueName: \"kubernetes.io/projected/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-kube-api-access-hqb46\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cnibin\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1609f9a3-09f2-487e-adc0-ed2436741052-cni-binary-copy\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-socket-dir-parent\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-k8s-cni-cncf-io\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1609f9a3-09f2-487e-adc0-ed2436741052-multus-daemon-config\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8vj\" (UniqueName: \"kubernetes.io/projected/74885f1b-db19-43da-9406-f294d2c898c4-kube-api-access-js8vj\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-cni-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-lib-modules\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-var-lib-kubelet\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74885f1b-db19-43da-9406-f294d2c898c4-etc-tuned\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74885f1b-db19-43da-9406-f294d2c898c4-tmp\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-system-cni-dir\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljcm\" (UniqueName: \"kubernetes.io/projected/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-kube-api-access-lljcm\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-system-cni-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-cni-multus\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-hostroot\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysconfig\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-sys\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.073964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-cni-bin\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.073990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-socket-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-sys-fs\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-hosts-file\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-cnibin\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074118    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wp875\""
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-conf-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-systemd\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-registration-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-device-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-kubelet\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2pw\" (UniqueName: \"kubernetes.io/projected/1609f9a3-09f2-487e-adc0-ed2436741052-kube-api-access-xh2pw\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysctl-conf\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074339    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-run\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-tmp-dir\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.075184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysctl-d\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-host\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074617    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074697    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.074790    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.074798    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.075075    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.076160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.075113    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 04:00:22.078169 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.078022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.078169 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.078103    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:22.080510 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.080491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:22.080606 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.080566    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:22.083132 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.083104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.086990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.086139    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 04:00:22.086990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.086258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 04:00:22.086990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.086981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cc2z2\""
Sep 08 04:00:22.087242 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.086995    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 04:00:22.095447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.095426    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2nxn"
Sep 08 04:00:22.106860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.106834    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2nxn"
Sep 08 04:00:22.157813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.157765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal" event={"ID":"9b957b5120b5ad80e4bc017af6c8f213","Type":"ContainerStarted","Data":"bf90897aae22704f8e0fa43c0622fbf2b96fcd4bae87764d02ea9b50a67738d0"}
Sep 08 04:00:22.159118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.159094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal" event={"ID":"bb5647e1cfe7a0d9b5f7192d30c19c7d","Type":"ContainerStarted","Data":"f4a1221d7af452a8fa7b5a6bbceafe328a57434cba6122414f82e0404eb67164"}
Sep 08 04:00:22.163243 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.163225    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 04:00:22.175650 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-ovn\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.175768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74885f1b-db19-43da-9406-f294d2c898c4-etc-tuned\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.175768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-serviceca\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.175768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.175768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2988\" (UniqueName: \"kubernetes.io/projected/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-kube-api-access-z2988\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.175768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-multus-certs\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-modprobe-d\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz744\" (UniqueName: \"kubernetes.io/projected/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-kube-api-access-lz744\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296v2\" (UniqueName: \"kubernetes.io/projected/ebc10fad-ac6b-4391-b3a6-987f6f033096-kube-api-access-296v2\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-etc-selinux\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqb46\" (UniqueName: \"kubernetes.io/projected/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-kube-api-access-hqb46\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cnibin\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.175988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1609f9a3-09f2-487e-adc0-ed2436741052-cni-binary-copy\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8vj\" (UniqueName: \"kubernetes.io/projected/74885f1b-db19-43da-9406-f294d2c898c4-kube-api-access-js8vj\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-kubelet\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-log-socket\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sd9b\" (UniqueName: \"kubernetes.io/projected/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-kube-api-access-4sd9b\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-systemd-units\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-cni-bin\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-cni-netd\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-cni-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-lib-modules\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-multus-certs\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74885f1b-db19-43da-9406-f294d2c898c4-tmp\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-run-netns\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-modprobe-d\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-node-log\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-env-overrides\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-etc-selinux\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovnkube-script-lib\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.176681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljcm\" (UniqueName: \"kubernetes.io/projected/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-kube-api-access-lljcm\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-cni-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-system-cni-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-cni-multus\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysconfig\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-system-cni-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cnibin\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-cni-multus\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-sys\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-lib-modules\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176774    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovnkube-config\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176816    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-sys\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovn-node-metrics-cert\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysconfig\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-host-slash\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-cni-bin\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.176983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-sys-fs\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.177479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-cni-bin\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-conf-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-sys-fs\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-systemd\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-conf-dir\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-systemd\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-host\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-run-ovn-kubernetes\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1609f9a3-09f2-487e-adc0-ed2436741052-cni-binary-copy\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0ab2194-4d54-4200-91fa-6d67c62cec94-kubelet-config\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-registration-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-device-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2pw\" (UniqueName: \"kubernetes.io/projected/1609f9a3-09f2-487e-adc0-ed2436741052-kube-api-access-xh2pw\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-registration-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-device-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-run\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f955d2b4-001f-4502-9c12-8fb652166bee-konnectivity-ca\") pod \"konnectivity-agent-btsdv\" (UID: \"f955d2b4-001f-4502-9c12-8fb652166bee\") " pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.178418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-slash\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-run\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-tmp-dir\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysctl-d\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-host\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-etc-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0ab2194-4d54-4200-91fa-6d67c62cec94-dbus\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-host\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-iptables-alerter-script\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysctl-d\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-etc-kubernetes\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-kubernetes\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzjs\" (UniqueName: \"kubernetes.io/projected/4d77fff3-3663-446a-af43-cfd608056894-kube-api-access-7kzjs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-tmp-dir\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.179165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-netns\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-etc-kubernetes\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-socket-dir-parent\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.177964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-netns\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-k8s-cni-cncf-io\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1609f9a3-09f2-487e-adc0-ed2436741052-multus-daemon-config\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-multus-socket-dir-parent\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-systemd\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-run-k8s-cni-cncf-io\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-var-lib-kubelet\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-kubernetes\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-system-cni-dir\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-var-lib-kubelet\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-hostroot\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.180016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-socket-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-system-cni-dir\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-hosts-file\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-cnibin\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f955d2b4-001f-4502-9c12-8fb652166bee-agent-certs\") pod \"konnectivity-agent-btsdv\" (UID: \"f955d2b4-001f-4502-9c12-8fb652166bee\") " pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-kubelet\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-hostroot\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysctl-conf\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-var-lib-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-cnibin\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178494    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-hosts-file\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1609f9a3-09f2-487e-adc0-ed2436741052-multus-daemon-config\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1609f9a3-09f2-487e-adc0-ed2436741052-host-var-lib-kubelet\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.180933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178691    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/74885f1b-db19-43da-9406-f294d2c898c4-etc-sysctl-conf\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.181719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-socket-dir\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.181719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.178937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.181719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.180619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74885f1b-db19-43da-9406-f294d2c898c4-tmp\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.181719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.180805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/74885f1b-db19-43da-9406-f294d2c898c4-etc-tuned\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.188462 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.188422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2988\" (UniqueName: \"kubernetes.io/projected/02b6f1af-7eab-47aa-a84d-66a045c2dbdf-kube-api-access-z2988\") pod \"node-resolver-rjjg2\" (UID: \"02b6f1af-7eab-47aa-a84d-66a045c2dbdf\") " pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.189117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.189093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqb46\" (UniqueName: \"kubernetes.io/projected/a2ed3b0b-6fe3-4c27-95c8-bfe123332925-kube-api-access-hqb46\") pod \"aws-ebs-csi-driver-node-r655v\" (UID: \"a2ed3b0b-6fe3-4c27-95c8-bfe123332925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.189488 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.189462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8vj\" (UniqueName: \"kubernetes.io/projected/74885f1b-db19-43da-9406-f294d2c898c4-kube-api-access-js8vj\") pod \"tuned-j65w8\" (UID: \"74885f1b-db19-43da-9406-f294d2c898c4\") " pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.190267 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.190239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljcm\" (UniqueName: \"kubernetes.io/projected/6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5-kube-api-access-lljcm\") pod \"multus-additional-cni-plugins-gts66\" (UID: \"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5\") " pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.192432 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.192406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2pw\" (UniqueName: \"kubernetes.io/projected/1609f9a3-09f2-487e-adc0-ed2436741052-kube-api-access-xh2pw\") pod \"multus-9nwc7\" (UID: \"1609f9a3-09f2-487e-adc0-ed2436741052\") " pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.264447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.264423    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 04:00:22.278750 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f955d2b4-001f-4502-9c12-8fb652166bee-agent-certs\") pod \"konnectivity-agent-btsdv\" (UID: \"f955d2b4-001f-4502-9c12-8fb652166bee\") " pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.278860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.278860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-var-lib-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.278860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-ovn\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.278860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-serviceca\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz744\" (UniqueName: \"kubernetes.io/projected/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-kube-api-access-lz744\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296v2\" (UniqueName: \"kubernetes.io/projected/ebc10fad-ac6b-4391-b3a6-987f6f033096-kube-api-access-296v2\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-kubelet\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-ovn\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-log-socket\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-var-lib-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.278967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sd9b\" (UniqueName: \"kubernetes.io/projected/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-kube-api-access-4sd9b\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-systemd-units\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-cni-bin\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-cni-netd\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-run-netns\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-node-log\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-env-overrides\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovnkube-script-lib\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovnkube-config\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovn-node-metrics-cert\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-host-slash\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-host\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-run-ovn-kubernetes\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0ab2194-4d54-4200-91fa-6d67c62cec94-kubelet-config\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f955d2b4-001f-4502-9c12-8fb652166bee-konnectivity-ca\") pod \"konnectivity-agent-btsdv\" (UID: \"f955d2b4-001f-4502-9c12-8fb652166bee\") " pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-slash\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-etc-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0ab2194-4d54-4200-91fa-6d67c62cec94-dbus\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.279669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-iptables-alerter-script\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzjs\" (UniqueName: \"kubernetes.io/projected/4d77fff3-3663-446a-af43-cfd608056894-kube-api-access-7kzjs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-systemd\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.279735    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.279839    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:22.779785338 +0000 UTC m=+3.138264882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-systemd-units\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-kubelet\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280042    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-cni-netd\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-run-netns\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-log-socket\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-node-log\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-cni-bin\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.280540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-env-overrides\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-slash\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f955d2b4-001f-4502-9c12-8fb652166bee-konnectivity-ca\") pod \"konnectivity-agent-btsdv\" (UID: \"f955d2b4-001f-4502-9c12-8fb652166bee\") " pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-run-systemd\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.279570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-serviceca\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-host-slash\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-host-run-ovn-kubernetes\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-host\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.280971    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.280989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ebc10fad-ac6b-4391-b3a6-987f6f033096-etc-openvswitch\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.281040    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:22.781021141 +0000 UTC m=+3.139500696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.281102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0ab2194-4d54-4200-91fa-6d67c62cec94-dbus\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.281109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovnkube-script-lib\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.281164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0ab2194-4d54-4200-91fa-6d67c62cec94-kubelet-config\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.281880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.281433    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovnkube-config\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.281880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.281670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-iptables-alerter-script\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.282376 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.282322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f955d2b4-001f-4502-9c12-8fb652166bee-agent-certs\") pod \"konnectivity-agent-btsdv\" (UID: \"f955d2b4-001f-4502-9c12-8fb652166bee\") " pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.283547 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.283525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ebc10fad-ac6b-4391-b3a6-987f6f033096-ovn-node-metrics-cert\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.294289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.294097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sd9b\" (UniqueName: \"kubernetes.io/projected/6af1c09d-0ba6-4998-891f-0c61ed0f0f35-kube-api-access-4sd9b\") pod \"node-ca-xjmjj\" (UID: \"6af1c09d-0ba6-4998-891f-0c61ed0f0f35\") " pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.296378 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.296357    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:22.296474 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.296384    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:22.296474 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.296397    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:22.296474 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.296455    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:22.796439861 +0000 UTC m=+3.154919404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:22.297471 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.297449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296v2\" (UniqueName: \"kubernetes.io/projected/ebc10fad-ac6b-4391-b3a6-987f6f033096-kube-api-access-296v2\") pod \"ovnkube-node-jp22z\" (UID: \"ebc10fad-ac6b-4391-b3a6-987f6f033096\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.303660 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.303637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz744\" (UniqueName: \"kubernetes.io/projected/6bb05d23-c5ea-4394-a6db-4b5bcabe3190-kube-api-access-lz744\") pod \"iptables-alerter-zkzhz\" (UID: \"6bb05d23-c5ea-4394-a6db-4b5bcabe3190\") " pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.303759 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.303705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzjs\" (UniqueName: \"kubernetes.io/projected/4d77fff3-3663-446a-af43-cfd608056894-kube-api-access-7kzjs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.368106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.368082    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v"
Sep 08 04:00:22.376986 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.376961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gts66"
Sep 08 04:00:22.385057 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.385030    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ed3b0b_6fe3_4c27_95c8_bfe123332925.slice/crio-31bcc6572db2f0c6a815ef66d3a22e9d7690224862901cd7ed8cda5cca823d60 WatchSource:0}: Error finding container 31bcc6572db2f0c6a815ef66d3a22e9d7690224862901cd7ed8cda5cca823d60: Status 404 returned error can't find the container with id 31bcc6572db2f0c6a815ef66d3a22e9d7690224862901cd7ed8cda5cca823d60
Sep 08 04:00:22.388579 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.388520    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6825e490_6fdf_4a0a_ae3f_2bbe9ea09fc5.slice/crio-8cfce3e65ec454939daec6d6a9c14cfc38b0899e65376b93e85ae354a79b6343 WatchSource:0}: Error finding container 8cfce3e65ec454939daec6d6a9c14cfc38b0899e65376b93e85ae354a79b6343: Status 404 returned error can't find the container with id 8cfce3e65ec454939daec6d6a9c14cfc38b0899e65376b93e85ae354a79b6343
Sep 08 04:00:22.393685 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.393659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rjjg2"
Sep 08 04:00:22.398639 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.398621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nwc7"
Sep 08 04:00:22.404733 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.404708    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b6f1af_7eab_47aa_a84d_66a045c2dbdf.slice/crio-9497772e1f5a252806d3557d49084af83b238e07edbb77b34f8ef491107cff71 WatchSource:0}: Error finding container 9497772e1f5a252806d3557d49084af83b238e07edbb77b34f8ef491107cff71: Status 404 returned error can't find the container with id 9497772e1f5a252806d3557d49084af83b238e07edbb77b34f8ef491107cff71
Sep 08 04:00:22.405585 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.405567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j65w8"
Sep 08 04:00:22.411043 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.411017    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1609f9a3_09f2_487e_adc0_ed2436741052.slice/crio-5ef6315dd6dba5a9d55956b9f6238610aba9a7df350d4b704e17713be9f522ba WatchSource:0}: Error finding container 5ef6315dd6dba5a9d55956b9f6238610aba9a7df350d4b704e17713be9f522ba: Status 404 returned error can't find the container with id 5ef6315dd6dba5a9d55956b9f6238610aba9a7df350d4b704e17713be9f522ba
Sep 08 04:00:22.413082 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.413055    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:22.416397 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.416371    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74885f1b_db19_43da_9406_f294d2c898c4.slice/crio-de31ce379fa57c404531119b47438bd629475896af6b6797dfaae7503019fe98 WatchSource:0}: Error finding container de31ce379fa57c404531119b47438bd629475896af6b6797dfaae7503019fe98: Status 404 returned error can't find the container with id de31ce379fa57c404531119b47438bd629475896af6b6797dfaae7503019fe98
Sep 08 04:00:22.419556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.419533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xjmjj"
Sep 08 04:00:22.424681 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.424653    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf955d2b4_001f_4502_9c12_8fb652166bee.slice/crio-5d901613f5a44ebf63acb5625e4da41ebc1ec185be801de719bd91d1297f1a6a WatchSource:0}: Error finding container 5d901613f5a44ebf63acb5625e4da41ebc1ec185be801de719bd91d1297f1a6a: Status 404 returned error can't find the container with id 5d901613f5a44ebf63acb5625e4da41ebc1ec185be801de719bd91d1297f1a6a
Sep 08 04:00:22.426805 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.426676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:22.429332 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.429286    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af1c09d_0ba6_4998_891f_0c61ed0f0f35.slice/crio-2c0744f7ece4f3218146502ec141dda3fdc3e547d8fe2f67fe8c18ae66f5d440 WatchSource:0}: Error finding container 2c0744f7ece4f3218146502ec141dda3fdc3e547d8fe2f67fe8c18ae66f5d440: Status 404 returned error can't find the container with id 2c0744f7ece4f3218146502ec141dda3fdc3e547d8fe2f67fe8c18ae66f5d440
Sep 08 04:00:22.434461 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.434435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zkzhz"
Sep 08 04:00:22.437736 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.437712    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc10fad_ac6b_4391_b3a6_987f6f033096.slice/crio-0ecd6406d0e1fb0096b9420bc589ad4ec22fba305971f300fd00a9977120901a WatchSource:0}: Error finding container 0ecd6406d0e1fb0096b9420bc589ad4ec22fba305971f300fd00a9977120901a: Status 404 returned error can't find the container with id 0ecd6406d0e1fb0096b9420bc589ad4ec22fba305971f300fd00a9977120901a
Sep 08 04:00:22.446362 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:22.446334    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb05d23_c5ea_4394_a6db_4b5bcabe3190.slice/crio-2b9b1a9025c27b73f41d00426f903096810fdb5b9f654e9e173805dd66782291 WatchSource:0}: Error finding container 2b9b1a9025c27b73f41d00426f903096810fdb5b9f654e9e173805dd66782291: Status 404 returned error can't find the container with id 2b9b1a9025c27b73f41d00426f903096810fdb5b9f654e9e173805dd66782291
Sep 08 04:00:22.782580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.782547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:22.782796 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.782608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:22.782796 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.782712    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:22.782796 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.782712    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:22.782796 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.782791    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:23.782757647 +0000 UTC m=+4.141237189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:22.782989 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.782810    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:23.782798011 +0000 UTC m=+4.141277570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:22.883690 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:22.883661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:22.883858 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.883800    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:22.883858 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.883820    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:22.883858 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.883839    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:22.883985 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:22.883886    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:23.883872527 +0000 UTC m=+4.242352066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:23.110175 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.110089    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 03:55:22 +0000 UTC" deadline="2028-02-15 00:53:39.634621243 +0000 UTC"
Sep 08 04:00:23.110175 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.110129    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12596h53m16.524496837s"
Sep 08 04:00:23.162069 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.162036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zkzhz" event={"ID":"6bb05d23-c5ea-4394-a6db-4b5bcabe3190","Type":"ContainerStarted","Data":"2b9b1a9025c27b73f41d00426f903096810fdb5b9f654e9e173805dd66782291"}
Sep 08 04:00:23.163334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.163289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xjmjj" event={"ID":"6af1c09d-0ba6-4998-891f-0c61ed0f0f35","Type":"ContainerStarted","Data":"2c0744f7ece4f3218146502ec141dda3fdc3e547d8fe2f67fe8c18ae66f5d440"}
Sep 08 04:00:23.164634 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.164609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j65w8" event={"ID":"74885f1b-db19-43da-9406-f294d2c898c4","Type":"ContainerStarted","Data":"de31ce379fa57c404531119b47438bd629475896af6b6797dfaae7503019fe98"}
Sep 08 04:00:23.165884 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.165858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nwc7" event={"ID":"1609f9a3-09f2-487e-adc0-ed2436741052","Type":"ContainerStarted","Data":"5ef6315dd6dba5a9d55956b9f6238610aba9a7df350d4b704e17713be9f522ba"}
Sep 08 04:00:23.167206 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.167176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerStarted","Data":"8cfce3e65ec454939daec6d6a9c14cfc38b0899e65376b93e85ae354a79b6343"}
Sep 08 04:00:23.168484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.168457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"0ecd6406d0e1fb0096b9420bc589ad4ec22fba305971f300fd00a9977120901a"}
Sep 08 04:00:23.169784 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.169757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-btsdv" event={"ID":"f955d2b4-001f-4502-9c12-8fb652166bee","Type":"ContainerStarted","Data":"5d901613f5a44ebf63acb5625e4da41ebc1ec185be801de719bd91d1297f1a6a"}
Sep 08 04:00:23.170970 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.170941    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rjjg2" event={"ID":"02b6f1af-7eab-47aa-a84d-66a045c2dbdf","Type":"ContainerStarted","Data":"9497772e1f5a252806d3557d49084af83b238e07edbb77b34f8ef491107cff71"}
Sep 08 04:00:23.172100 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.172077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v" event={"ID":"a2ed3b0b-6fe3-4c27-95c8-bfe123332925","Type":"ContainerStarted","Data":"31bcc6572db2f0c6a815ef66d3a22e9d7690224862901cd7ed8cda5cca823d60"}
Sep 08 04:00:23.791481 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.791226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:23.791699 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.791535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:23.791699 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.791369    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:23.791699 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.791655    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:25.79163192 +0000 UTC m=+6.150111475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:23.791699 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.791676    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:23.791956 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.791727    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:25.791711724 +0000 UTC m=+6.150191267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:23.892799 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:23.892218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:23.892799 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.892364    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:23.892799 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.892385    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:23.892799 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.892399    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:23.892799 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:23.892464    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:25.892445813 +0000 UTC m=+6.250925359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:24.110858 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:24.110753    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 03:55:22 +0000 UTC" deadline="2028-05-21 20:07:15.724148744 +0000 UTC"
Sep 08 04:00:24.110858 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:24.110798    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14920h6m51.613355199s"
Sep 08 04:00:24.157230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:24.154093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:24.157230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:24.154098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:24.157230 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:24.154243    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:24.157230 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:24.154377    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:24.157230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:24.154436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:24.157230 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:24.154503    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:24.183371 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:24.183338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal" event={"ID":"bb5647e1cfe7a0d9b5f7192d30c19c7d","Type":"ContainerStarted","Data":"fcf5fcfd015976860d68131540f1d81ea248b0d775fd53426e2833765338dc1d"}
Sep 08 04:00:25.806286 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:25.806246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:25.806775 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:25.806331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:25.806775 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.806448    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:25.806775 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.806504    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:29.806486117 +0000 UTC m=+10.164965661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:25.806975 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.806926    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:25.807036 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.806978    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:29.806962537 +0000 UTC m=+10.165442081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:25.906987 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:25.906947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:25.907156 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.907107    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:25.907156 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.907139    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:25.907156 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.907155    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:25.907340 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:25.907215    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:29.907193941 +0000 UTC m=+10.265673479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:26.157992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:26.156316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:26.157992 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:26.156450    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:26.157992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:26.156878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:26.157992 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:26.157005    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:26.157992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:26.157832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:26.157992 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:26.157949    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:28.154911 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:28.154878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:28.154911 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:28.154905    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:28.155484 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:28.155022    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:28.155484 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:28.155102    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:28.155484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:28.155142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:28.155484 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:28.155263    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:29.840417 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:29.840178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:29.840417 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:29.840254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:29.840417 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.840346    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:29.840417 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.840383    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:29.840417 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.840415    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:37.840394239 +0000 UTC m=+18.198873795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:29.841304 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.840442    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:37.840425145 +0000 UTC m=+18.198904684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:29.940673 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:29.940643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:29.940904 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.940818    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:29.940904 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.940842    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:29.940904 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.940855    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:29.941073 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:29.940909    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:37.940892598 +0000 UTC m=+18.299372154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:30.154774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:30.154656    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:30.154774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:30.154759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:30.154980 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:30.154776    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:30.154980 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:30.154872    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:30.154980 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:30.154901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:30.154980 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:30.154971    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:32.154425 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:32.154393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:32.154425 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:32.154420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:32.155035 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:32.154401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:32.155035 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:32.154524    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:32.155035 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:32.154603    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:32.155035 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:32.154695    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:34.154444 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:34.154357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:34.154444 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:34.154403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:34.154939 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:34.154363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:34.154939 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:34.154486    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:34.154939 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:34.154580    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:34.154939 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:34.154649    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:36.153969 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:36.153936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:36.154404 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:36.154061    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:36.154404 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:36.154070    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:36.154404 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:36.154107    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:36.154404 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:36.154179    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:36.154404 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:36.154253    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:37.897329 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:37.897245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:37.897329 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:37.897316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:37.897790 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.897413    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:37.897790 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.897481    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:53.897464107 +0000 UTC m=+34.255943649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:37.897790 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.897413    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:37.897790 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.897597    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:53.897574539 +0000 UTC m=+34.256054086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:37.998078 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:37.998046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:37.998234 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.998213    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:37.998286 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.998243    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:37.998286 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.998254    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:37.998375 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:37.998304    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:53.998289479 +0000 UTC m=+34.356769021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:38.154148 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:38.154068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:38.154148 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:38.154082    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:38.154333 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:38.154177    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:38.154333 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:38.154246    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:38.154333 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:38.154276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:38.154449 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:38.154370    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:40.155223 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:40.155182    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:40.155828 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:40.155296    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:40.155828 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:40.155398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:40.155828 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:40.155505    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:40.155828 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:40.155538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:40.155828 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:40.155621    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:42.154373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.154152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:42.155047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.154170    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:42.155047 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:42.154406    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:42.155047 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:42.154506    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:42.155047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.154224    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:42.155047 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:42.154601    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:42.223884 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.223858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xjmjj" event={"ID":"6af1c09d-0ba6-4998-891f-0c61ed0f0f35","Type":"ContainerStarted","Data":"2c6aa57796efc06dc618dfc619fc23dd216917f654129a8e8b03f4bb3ddb49aa"}
Sep 08 04:00:42.225361 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.225338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j65w8" event={"ID":"74885f1b-db19-43da-9406-f294d2c898c4","Type":"ContainerStarted","Data":"b403a33cd4a8f6ed9645fa54dfea200386e84901f6bedf7f5fa836a50efc4b78"}
Sep 08 04:00:42.226713 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.226692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nwc7" event={"ID":"1609f9a3-09f2-487e-adc0-ed2436741052","Type":"ContainerStarted","Data":"9c0d3b2a34d07cad79fa2c6b325baafca2c94895ec81e81213e7978f29ad94ae"}
Sep 08 04:00:42.228263 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.228241    2583 generic.go:356] "Generic (PLEG): container finished" podID="6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5" containerID="a46d31851f26e2d3dd375bb17110f38a4d367d91cb5b73e1d2a2dc0126fdb959" exitCode=0
Sep 08 04:00:42.228358 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.228326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerDied","Data":"a46d31851f26e2d3dd375bb17110f38a4d367d91cb5b73e1d2a2dc0126fdb959"}
Sep 08 04:00:42.229970 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.229941    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b957b5120b5ad80e4bc017af6c8f213" containerID="40e97fcdcfcfb31754b9bb6b0a00f151b7601be26471c3e865035b895b86ee17" exitCode=0
Sep 08 04:00:42.230061 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.230011    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal" event={"ID":"9b957b5120b5ad80e4bc017af6c8f213","Type":"ContainerDied","Data":"40e97fcdcfcfb31754b9bb6b0a00f151b7601be26471c3e865035b895b86ee17"}
Sep 08 04:00:42.233251 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.233231    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"f61cb8f0295767f6b7075c1cd449fe4b629ab083edc94b6513d6eb96f905ebbe"}
Sep 08 04:00:42.233337 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.233300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"b214997612998861a0140ffab972f7e41568af822112cc27c9031b988f6bbc00"}
Sep 08 04:00:42.233408 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.233341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"c9d211c5b21b9e885484d4410d5c2dc577637b46d6739ff2807602436c672a09"}
Sep 08 04:00:42.233408 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.233378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"509b81cb67f344b3abe56bae732516788975c2b442eff835397cf177444dc08b"}
Sep 08 04:00:42.233483 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.233412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"966516fec06a4e24cba021468d57b04db807d71081e324b52f8c4aeb8a5cbbef"}
Sep 08 04:00:42.233483 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.233447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"a0d58171378e2921c9394b2ecfbf0cb4d1a0fd8016cbe5ab2544b0ef97a0a26f"}
Sep 08 04:00:42.234681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.234661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-btsdv" event={"ID":"f955d2b4-001f-4502-9c12-8fb652166bee","Type":"ContainerStarted","Data":"07b60d280190c538d954e86106dfd496faf451c81c6c2234bec09a3e3292dd8a"}
Sep 08 04:00:42.235980 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.235956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rjjg2" event={"ID":"02b6f1af-7eab-47aa-a84d-66a045c2dbdf","Type":"ContainerStarted","Data":"2426995725d2b4195dc40fa4b0ff45834307a6ec63e299fbe96d70aba24d80aa"}
Sep 08 04:00:42.237192 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.237175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v" event={"ID":"a2ed3b0b-6fe3-4c27-95c8-bfe123332925","Type":"ContainerStarted","Data":"6724785e991af81c2c153ec2b4b82412f9fb1c726208efdb71d558bd90bb4957"}
Sep 08 04:00:42.239820 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.239786    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.ec2.internal" podStartSLOduration=20.239775248 podStartE2EDuration="20.239775248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:24.203673784 +0000 UTC m=+4.562153349" watchObservedRunningTime="2026-09-08 04:00:42.239775248 +0000 UTC m=+22.598254808"
Sep 08 04:00:42.239990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.239973    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xjmjj" podStartSLOduration=2.454612855 podStartE2EDuration="21.239968454s" totalImagesPullingTime="18.785355599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:42.239339383 +0000 UTC m=+22.597818946" watchObservedRunningTime="2026-09-08 04:00:42.239968454 +0000 UTC m=+22.598448046"
Sep 08 04:00:42.276696 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.276650    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rjjg2" podStartSLOduration=7.180751649 podStartE2EDuration="22.276634979s" totalImagesPullingTime="15.09588333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:42.276358291 +0000 UTC m=+22.634837852" watchObservedRunningTime="2026-09-08 04:00:42.276634979 +0000 UTC m=+22.635114540"
Sep 08 04:00:42.350214 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.350170    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j65w8" podStartSLOduration=2.536554651 podStartE2EDuration="21.350157788s" totalImagesPullingTime="18.813603137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:42.349789156 +0000 UTC m=+22.708268712" watchObservedRunningTime="2026-09-08 04:00:42.350157788 +0000 UTC m=+22.708637347"
Sep 08 04:00:42.350395 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.350370    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9nwc7" podStartSLOduration=2.529152743 podStartE2EDuration="21.350363616s" totalImagesPullingTime="18.821210873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:42.304349509 +0000 UTC m=+22.662829068" watchObservedRunningTime="2026-09-08 04:00:42.350363616 +0000 UTC m=+22.708843177"
Sep 08 04:00:42.365759 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.365709    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-btsdv" podStartSLOduration=2.682049177 podStartE2EDuration="21.36569768s" totalImagesPullingTime="18.683648503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:42.365533604 +0000 UTC m=+22.724013165" watchObservedRunningTime="2026-09-08 04:00:42.36569768 +0000 UTC m=+22.724177277"
Sep 08 04:00:42.443723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.443700    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:42.444172 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:42.444145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:43.241085 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:43.241048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zkzhz" event={"ID":"6bb05d23-c5ea-4394-a6db-4b5bcabe3190","Type":"ContainerStarted","Data":"ba435057adc26210be4a597a2942e1433e99720f19a3492f99601d5033b92cfe"}
Sep 08 04:00:43.243246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:43.243104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal" event={"ID":"9b957b5120b5ad80e4bc017af6c8f213","Type":"ContainerStarted","Data":"dba9fa0a7e2f304a669a0ee2804b6fb7da71b8ecbe1dbe3112c224c8abbbc1e7"}
Sep 08 04:00:43.243993 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:43.243973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:43.244554 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:43.244511    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-btsdv"
Sep 08 04:00:43.258232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:43.258198    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zkzhz" podStartSLOduration=3.489674384 podStartE2EDuration="22.258184377s" totalImagesPullingTime="18.768509993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:43.257871755 +0000 UTC m=+23.616351295" watchObservedRunningTime="2026-09-08 04:00:43.258184377 +0000 UTC m=+23.616663938"
Sep 08 04:00:43.294340 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:43.294285    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.ec2.internal" podStartSLOduration=21.294265208 podStartE2EDuration="21.294265208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:43.274130207 +0000 UTC m=+23.632609768" watchObservedRunningTime="2026-09-08 04:00:43.294265208 +0000 UTC m=+23.652744767"
Sep 08 04:00:44.153690 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:44.153657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:44.153819 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:44.153662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:44.153819 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:44.153798    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:44.153934 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:44.153662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:44.153934 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:44.153886    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:44.154031 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:44.153965    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:44.248831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:44.248799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"4f292994eff1f302644cc06763c43c6ca57b5215886ed4fd7624ee25e67438a2"}
Sep 08 04:00:46.157163 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:46.156959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:46.157526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:46.157004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:46.157526 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:46.157259    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:46.157526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:46.157039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:46.157526 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:46.157338    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:46.157526 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:46.157388    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:47.756622 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:47.755453    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 04:00:48.124596 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.124483    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T04:00:47.755471142Z","UUID":"420ba46f-aeba-40d1-a1df-f439c5c0b262","Handler":null,"Name":"","Endpoint":""}
Sep 08 04:00:48.125986 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.125969    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 04:00:48.126069 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.125996    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 04:00:48.154523 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.154494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:48.154523 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.154507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:48.154674 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.154494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:48.154674 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:48.154586    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:48.154793 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:48.154701    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:48.154850 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:48.154808    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:48.258202 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.258178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v" event={"ID":"a2ed3b0b-6fe3-4c27-95c8-bfe123332925","Type":"ContainerStarted","Data":"fed4195ca8acc7f3eb53de74479dff68333e5ff73d4787ede7113c1289889204"}
Sep 08 04:00:48.260022 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.260003    2583 generic.go:356] "Generic (PLEG): container finished" podID="6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5" containerID="bf301eb553706400e5b235eaebceb471d8803e33e506fc6c424fcc8101db119f" exitCode=0
Sep 08 04:00:48.260103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.260086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerDied","Data":"bf301eb553706400e5b235eaebceb471d8803e33e506fc6c424fcc8101db119f"}
Sep 08 04:00:48.264016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.263996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" event={"ID":"ebc10fad-ac6b-4391-b3a6-987f6f033096","Type":"ContainerStarted","Data":"1fe8643ffa84399675554bd8810325cdf046992a7ecbd3d8753cbadb8feb1215"}
Sep 08 04:00:48.264250 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.264227    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:48.264334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.264274    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:48.278836 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.278818    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:48.311578 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:48.311542    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z" podStartSLOduration=8.473735865 podStartE2EDuration="27.31153061s" totalImagesPullingTime="18.837794745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:48.311485009 +0000 UTC m=+28.669964569" watchObservedRunningTime="2026-09-08 04:00:48.31153061 +0000 UTC m=+28.670010170"
Sep 08 04:00:49.266433 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.266406    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:49.280123 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.280098    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:00:49.593535 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.593480    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lrg6q"]
Sep 08 04:00:49.593654 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.593581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:49.593709 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:49.593685    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:49.597174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.597150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bw44h"]
Sep 08 04:00:49.597267 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.597234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:49.597345 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:49.597323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:49.597882 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.597865    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kj9xx"]
Sep 08 04:00:49.597953 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:49.597938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:49.598031 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:49.598013    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:50.271290 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:50.271190    2583 generic.go:356] "Generic (PLEG): container finished" podID="6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5" containerID="d677f3e5ae21a14a5d35f1c33e47075703bc2a8b8b8ac6501f6652589224bd05" exitCode=0
Sep 08 04:00:50.271290 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:50.271249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerDied","Data":"d677f3e5ae21a14a5d35f1c33e47075703bc2a8b8b8ac6501f6652589224bd05"}
Sep 08 04:00:51.153789 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:51.153619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:51.153945 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:51.153644    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:51.153945 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:51.153654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:51.154049 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:51.153952    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:51.154049 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:51.153990    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:51.154049 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:51.153851    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:51.275626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:51.275600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v" event={"ID":"a2ed3b0b-6fe3-4c27-95c8-bfe123332925","Type":"ContainerStarted","Data":"94d6fcb6b36228ef854600435a1783f1da2736c75d00341adf96078889f21608"}
Sep 08 04:00:51.304062 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:51.304024    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r655v" podStartSLOduration=3.518120059 podStartE2EDuration="31.304012281s" totalImagesPullingTime="27.785892222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:51.303873512 +0000 UTC m=+31.662353071" watchObservedRunningTime="2026-09-08 04:00:51.304012281 +0000 UTC m=+31.662491858"
Sep 08 04:00:53.154123 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.154090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:53.154406 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.154122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:53.154406 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.154198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:53.154406 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.154205    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bw44h" podUID="2d20074d-f010-4796-ae28-de01ed79d714"
Sep 08 04:00:53.154406 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.154272    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrg6q" podUID="b0ab2194-4d54-4200-91fa-6d67c62cec94"
Sep 08 04:00:53.154406 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.154360    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj9xx" podUID="4d77fff3-3663-446a-af43-cfd608056894"
Sep 08 04:00:53.282082 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.282059    2583 generic.go:356] "Generic (PLEG): container finished" podID="6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5" containerID="b5d5aa6cbb4ae17b5b8f7b3414a2d3798913534c066f06765c4f0528a20634e9" exitCode=0
Sep 08 04:00:53.282182 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.282115    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerDied","Data":"b5d5aa6cbb4ae17b5b8f7b3414a2d3798913534c066f06765c4f0528a20634e9"}
Sep 08 04:00:53.914045 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.913970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:53.914045 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:53.914011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:53.914219 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.914097    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:53.914219 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.914098    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:53.914219 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.914141    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret podName:b0ab2194-4d54-4200-91fa-6d67c62cec94 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:25.914129766 +0000 UTC m=+66.272609303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret") pod "global-pull-secret-syncer-lrg6q" (UID: "b0ab2194-4d54-4200-91fa-6d67c62cec94") : object "kube-system"/"original-pull-secret" not registered
Sep 08 04:00:53.914219 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:53.914155    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs podName:4d77fff3-3663-446a-af43-cfd608056894 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:25.914149322 +0000 UTC m=+66.272628859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs") pod "network-metrics-daemon-kj9xx" (UID: "4d77fff3-3663-446a-af43-cfd608056894") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 04:00:54.014831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.014807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:54.014951 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:54.014939    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 04:00:54.014951 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:54.014952    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 04:00:54.015028 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:54.014960    2583 projected.go:196] Error preparing data for projected volume kube-api-access-dz2c2 for pod openshift-network-diagnostics/network-check-target-bw44h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:54.015028 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:54.014998    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2 podName:2d20074d-f010-4796-ae28-de01ed79d714 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.014985722 +0000 UTC m=+66.373465264 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz2c2" (UniqueName: "kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2") pod "network-check-target-bw44h" (UID: "2d20074d-f010-4796-ae28-de01ed79d714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 04:00:54.892146 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.892123    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-111.ec2.internal" event="NodeReady"
Sep 08 04:00:54.892496 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.892243    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 04:00:54.927254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.927231    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f6b748b67-h2bdx"]
Sep 08 04:00:54.941084 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.940975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:54.943542 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.943505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 04:00:54.943542 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.943509    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 04:00:54.943542 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.943512    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qx7pf\""
Sep 08 04:00:54.943782 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.943574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 04:00:54.954399 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.954370    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 04:00:54.954576 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.954559    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6b748b67-h2bdx"]
Sep 08 04:00:54.954961 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.954936    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m5h4q"]
Sep 08 04:00:54.973214 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.973195    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gh8zx"]
Sep 08 04:00:54.973330 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.973315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:54.976009 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.975987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 04:00:54.976124 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.976065    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26sdq\""
Sep 08 04:00:54.976124 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.976069    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 04:00:54.985525 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.985507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5h4q"]
Sep 08 04:00:54.985635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.985538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gh8zx"]
Sep 08 04:00:54.985635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.985615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:54.987962 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.987935    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 04:00:54.988068 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.987996    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 04:00:54.988126 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.987943    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9nxct\""
Sep 08 04:00:54.988181 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:54.988173    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 04:00:55.021732 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.021732 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73078221-e155-4445-94e0-533af6d4b81a-ca-trust-extracted\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.021899 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-trusted-ca\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.021899 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-registry-certificates\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.021899 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-installation-pull-secrets\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.021899 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-bound-sa-token\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.022105 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrps\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-kube-api-access-jnrps\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.022105 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.021963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-image-registry-private-configuration\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.122606 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-image-registry-private-configuration\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.122730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.122730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73078221-e155-4445-94e0-533af6d4b81a-ca-trust-extracted\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.122730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.122730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-trusted-ca\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.122730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e4a1f-44cd-45a0-ba63-82faba49d733-config-volume\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.123017 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-registry-certificates\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.123017 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-installation-pull-secrets\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.123017 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.122811    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 04:00:55.123017 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.122876    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b748b67-h2bdx: secret "image-registry-tls" not found
Sep 08 04:00:55.123017 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.122874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6e4a1f-44cd-45a0-ba63-82faba49d733-tmp-dir\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.123017 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.122946    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls podName:73078221-e155-4445-94e0-533af6d4b81a nodeName:}" failed. No retries permitted until 2026-09-08 04:00:55.622926816 +0000 UTC m=+35.981406369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls") pod "image-registry-5f6b748b67-h2bdx" (UID: "73078221-e155-4445-94e0-533af6d4b81a") : secret "image-registry-tls" not found
Sep 08 04:00:55.123322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmxn\" (UniqueName: \"kubernetes.io/projected/4a6e4a1f-44cd-45a0-ba63-82faba49d733-kube-api-access-dcmxn\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.123322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73078221-e155-4445-94e0-533af6d4b81a-ca-trust-extracted\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.123322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-bound-sa-token\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.123322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrps\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-kube-api-access-jnrps\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.123322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0-cert\") pod \"ingress-canary-gh8zx\" (UID: \"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0\") " pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.123322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kl89\" (UniqueName: \"kubernetes.io/projected/bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0-kube-api-access-2kl89\") pod \"ingress-canary-gh8zx\" (UID: \"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0\") " pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.123597 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.123555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-registry-certificates\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.124197 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.124016    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-trusted-ca\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.127336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.127315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-image-registry-private-configuration\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.127435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.127396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-installation-pull-secrets\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.133556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.133538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-bound-sa-token\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.133674 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.133652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrps\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-kube-api-access-jnrps\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.153484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.153457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:00:55.153569 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.153550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:00:55.153627 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.153577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:00:55.156542 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.156507    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 04:00:55.156636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.156540    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 04:00:55.156636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.156557    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-blqwr\""
Sep 08 04:00:55.156914 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.156896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 04:00:55.157040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.156895    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 04:00:55.157040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.156898    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f9485\""
Sep 08 04:00:55.224464 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0-cert\") pod \"ingress-canary-gh8zx\" (UID: \"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0\") " pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.224549 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kl89\" (UniqueName: \"kubernetes.io/projected/bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0-kube-api-access-2kl89\") pod \"ingress-canary-gh8zx\" (UID: \"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0\") " pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.224588 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.224588 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e4a1f-44cd-45a0-ba63-82faba49d733-config-volume\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.224683 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6e4a1f-44cd-45a0-ba63-82faba49d733-tmp-dir\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.224683 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmxn\" (UniqueName: \"kubernetes.io/projected/4a6e4a1f-44cd-45a0-ba63-82faba49d733-kube-api-access-dcmxn\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.224803 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.224685    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 04:00:55.224803 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.224776    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls podName:4a6e4a1f-44cd-45a0-ba63-82faba49d733 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:55.72475542 +0000 UTC m=+36.083234977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls") pod "dns-default-m5h4q" (UID: "4a6e4a1f-44cd-45a0-ba63-82faba49d733") : secret "dns-default-metrics-tls" not found
Sep 08 04:00:55.224969 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.224941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6e4a1f-44cd-45a0-ba63-82faba49d733-tmp-dir\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.225122 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.225106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e4a1f-44cd-45a0-ba63-82faba49d733-config-volume\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.226671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.226654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0-cert\") pod \"ingress-canary-gh8zx\" (UID: \"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0\") " pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.232946 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.232923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kl89\" (UniqueName: \"kubernetes.io/projected/bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0-kube-api-access-2kl89\") pod \"ingress-canary-gh8zx\" (UID: \"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0\") " pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.233021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.232926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmxn\" (UniqueName: \"kubernetes.io/projected/4a6e4a1f-44cd-45a0-ba63-82faba49d733-kube-api-access-dcmxn\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.295133 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.295082    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gh8zx"
Sep 08 04:00:55.606898 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.606630    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gh8zx"]
Sep 08 04:00:55.610899 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:00:55.610873    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb0eb0f_c7aa_4858_b2bb_d779782a7ad0.slice/crio-ca5f7c177a91e051dedd1b6853e0cabdaa893476980322e09c039a57d7a0346d WatchSource:0}: Error finding container ca5f7c177a91e051dedd1b6853e0cabdaa893476980322e09c039a57d7a0346d: Status 404 returned error can't find the container with id ca5f7c177a91e051dedd1b6853e0cabdaa893476980322e09c039a57d7a0346d
Sep 08 04:00:55.628455 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.628435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:55.628586 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.628556    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 04:00:55.628586 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.628580    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b748b67-h2bdx: secret "image-registry-tls" not found
Sep 08 04:00:55.628757 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.628661    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls podName:73078221-e155-4445-94e0-533af6d4b81a nodeName:}" failed. No retries permitted until 2026-09-08 04:00:56.628646082 +0000 UTC m=+36.987125632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls") pod "image-registry-5f6b748b67-h2bdx" (UID: "73078221-e155-4445-94e0-533af6d4b81a") : secret "image-registry-tls" not found
Sep 08 04:00:55.729699 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:55.729669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:55.729857 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.729801    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 04:00:55.729857 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:55.729851    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls podName:4a6e4a1f-44cd-45a0-ba63-82faba49d733 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:56.729835301 +0000 UTC m=+37.088314851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls") pod "dns-default-m5h4q" (UID: "4a6e4a1f-44cd-45a0-ba63-82faba49d733") : secret "dns-default-metrics-tls" not found
Sep 08 04:00:56.293591 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:56.292297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gh8zx" event={"ID":"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0","Type":"ContainerStarted","Data":"ca5f7c177a91e051dedd1b6853e0cabdaa893476980322e09c039a57d7a0346d"}
Sep 08 04:00:56.635724 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:56.635645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:56.635891 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:56.635832    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 04:00:56.635891 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:56.635854    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b748b67-h2bdx: secret "image-registry-tls" not found
Sep 08 04:00:56.636007 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:56.635919    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls podName:73078221-e155-4445-94e0-533af6d4b81a nodeName:}" failed. No retries permitted until 2026-09-08 04:00:58.635899816 +0000 UTC m=+38.994379357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls") pod "image-registry-5f6b748b67-h2bdx" (UID: "73078221-e155-4445-94e0-533af6d4b81a") : secret "image-registry-tls" not found
Sep 08 04:00:56.736985 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:56.736951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:56.737141 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:56.737123    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 04:00:56.737212 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:56.737192    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls podName:4a6e4a1f-44cd-45a0-ba63-82faba49d733 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:58.737175782 +0000 UTC m=+39.095655328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls") pod "dns-default-m5h4q" (UID: "4a6e4a1f-44cd-45a0-ba63-82faba49d733") : secret "dns-default-metrics-tls" not found
Sep 08 04:00:58.654695 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:58.654659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:00:58.655272 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:58.654837    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 04:00:58.655272 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:58.654860    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b748b67-h2bdx: secret "image-registry-tls" not found
Sep 08 04:00:58.655272 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:58.654917    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls podName:73078221-e155-4445-94e0-533af6d4b81a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:02.654902429 +0000 UTC m=+43.013381967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls") pod "image-registry-5f6b748b67-h2bdx" (UID: "73078221-e155-4445-94e0-533af6d4b81a") : secret "image-registry-tls" not found
Sep 08 04:00:58.755518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:00:58.755478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:00:58.755668 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:58.755628    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 04:00:58.755733 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:00:58.755695    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls podName:4a6e4a1f-44cd-45a0-ba63-82faba49d733 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:02.755679454 +0000 UTC m=+43.114158992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls") pod "dns-default-m5h4q" (UID: "4a6e4a1f-44cd-45a0-ba63-82faba49d733") : secret "dns-default-metrics-tls" not found
Sep 08 04:01:00.304676 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:00.304643    2583 generic.go:356] "Generic (PLEG): container finished" podID="6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5" containerID="9e99130c7958d5c044f1eb9032855c21c58f129abb3d5dd3be4c439a02781326" exitCode=0
Sep 08 04:01:00.305189 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:00.304708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerDied","Data":"9e99130c7958d5c044f1eb9032855c21c58f129abb3d5dd3be4c439a02781326"}
Sep 08 04:01:00.306167 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:00.306140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gh8zx" event={"ID":"bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0","Type":"ContainerStarted","Data":"fea2a25d7068282559d9a1b379c2bac052cdd160fdbd6e1d778b362bd9805613"}
Sep 08 04:01:00.349404 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:00.349357    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gh8zx" podStartSLOduration=2.668611724 podStartE2EDuration="6.349339568s" totalImagesPullingTime="3.680727844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:00.34909466 +0000 UTC m=+40.707574226" watchObservedRunningTime="2026-09-08 04:01:00.349339568 +0000 UTC m=+40.707819129"
Sep 08 04:01:01.312087 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:01.312056    2583 generic.go:356] "Generic (PLEG): container finished" podID="6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5" containerID="e908185193fa66354accbd812db258726d9de66a36a9be1991f317fe2fe9c686" exitCode=0
Sep 08 04:01:01.312457 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:01.312135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerDied","Data":"e908185193fa66354accbd812db258726d9de66a36a9be1991f317fe2fe9c686"}
Sep 08 04:01:02.317204 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:02.317171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gts66" event={"ID":"6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5","Type":"ContainerStarted","Data":"3b65d47f34284c6089a3c87af164291031b3ceaf9d9ab3472fc6a55aec3da1d6"}
Sep 08 04:01:02.338584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:02.338525    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gts66" podStartSLOduration=8.640165004 podStartE2EDuration="42.338509007s" totalImagesPullingTime="33.698344003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:02.337243526 +0000 UTC m=+42.695723114" watchObservedRunningTime="2026-09-08 04:01:02.338509007 +0000 UTC m=+42.696988567"
Sep 08 04:01:02.679780 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:02.679720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:02.679887 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:02.679861    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 04:01:02.679887 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:02.679878    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b748b67-h2bdx: secret "image-registry-tls" not found
Sep 08 04:01:02.679955 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:02.679923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls podName:73078221-e155-4445-94e0-533af6d4b81a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:10.6799099 +0000 UTC m=+51.038389439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls") pod "image-registry-5f6b748b67-h2bdx" (UID: "73078221-e155-4445-94e0-533af6d4b81a") : secret "image-registry-tls" not found
Sep 08 04:01:02.780257 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:02.780235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:02.780365 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:02.780321    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 04:01:02.780402 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:02.780377    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls podName:4a6e4a1f-44cd-45a0-ba63-82faba49d733 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:10.780366178 +0000 UTC m=+51.138845717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls") pod "dns-default-m5h4q" (UID: "4a6e4a1f-44cd-45a0-ba63-82faba49d733") : secret "dns-default-metrics-tls" not found
Sep 08 04:01:04.075092 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.075060    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"]
Sep 08 04:01:04.118979 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.118953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"]
Sep 08 04:01:04.119096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.118984    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-56mr8"]
Sep 08 04:01:04.119096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.119072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"
Sep 08 04:01:04.121623 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.121601    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 04:01:04.121759 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.121621    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 04:01:04.121759 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.121693    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jvk2d\""
Sep 08 04:01:04.137893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.137871    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-jpbbn"]
Sep 08 04:01:04.138004 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.137987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8"
Sep 08 04:01:04.140023 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.140005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bjfs2\""
Sep 08 04:01:04.150243 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.150227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-56mr8"]
Sep 08 04:01:04.150317 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.150251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-jpbbn"]
Sep 08 04:01:04.150355 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.150324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.153210 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.153187    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 04:01:04.153308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.153229    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 04:01:04.153366 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.153333    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xzprc\""
Sep 08 04:01:04.153423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.153383    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 04:01:04.153768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.153750    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 04:01:04.158056 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.158036    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 04:01:04.188143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.188121    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8rqfd"]
Sep 08 04:01:04.208338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.208317    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8rqfd"]
Sep 08 04:01:04.208430 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.208397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.212113 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.212088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 04:01:04.212448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.212430    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 04:01:04.213004 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.212976    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 04:01:04.213077 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.213013    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 04:01:04.213243 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.213228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lzpmw\""
Sep 08 04:01:04.218018 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.218000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 04:01:04.289925 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.289907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f092e55-100b-4423-b95f-be552520d428-tmp\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.290012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.289936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6931a442-2287-43dd-b61f-93ffa6e89c5b-serving-cert\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.290012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.289953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6931a442-2287-43dd-b61f-93ffa6e89c5b-trusted-ca\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.290012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.289969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94d4b\" (UniqueName: \"kubernetes.io/projected/6931a442-2287-43dd-b61f-93ffa6e89c5b-kube-api-access-94d4b\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.290012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.289983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f092e55-100b-4423-b95f-be552520d428-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.290161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqhl\" (UniqueName: \"kubernetes.io/projected/6f092e55-100b-4423-b95f-be552520d428-kube-api-access-5fqhl\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.290161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddg27\" (UniqueName: \"kubernetes.io/projected/cb500cea-25dc-43d9-af5b-76a809317a4e-kube-api-access-ddg27\") pod \"network-check-source-685456655f-56mr8\" (UID: \"cb500cea-25dc-43d9-af5b-76a809317a4e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8"
Sep 08 04:01:04.290161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6931a442-2287-43dd-b61f-93ffa6e89c5b-config\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.290161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f092e55-100b-4423-b95f-be552520d428-serving-cert\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.290161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmq5\" (UniqueName: \"kubernetes.io/projected/2e181c82-660a-47df-b138-a3b81c374946-kube-api-access-4qmq5\") pod \"volume-data-source-validator-695c895bd6-r2vsj\" (UID: \"2e181c82-660a-47df-b138-a3b81c374946\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"
Sep 08 04:01:04.290161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6f092e55-100b-4423-b95f-be552520d428-snapshots\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.290345 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.290170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f092e55-100b-4423-b95f-be552520d428-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391111 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6f092e55-100b-4423-b95f-be552520d428-snapshots\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391111 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f092e55-100b-4423-b95f-be552520d428-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391214 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f092e55-100b-4423-b95f-be552520d428-tmp\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391214 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6931a442-2287-43dd-b61f-93ffa6e89c5b-serving-cert\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.391292 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6931a442-2287-43dd-b61f-93ffa6e89c5b-trusted-ca\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.391292 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94d4b\" (UniqueName: \"kubernetes.io/projected/6931a442-2287-43dd-b61f-93ffa6e89c5b-kube-api-access-94d4b\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.391292 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f092e55-100b-4423-b95f-be552520d428-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391457 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqhl\" (UniqueName: \"kubernetes.io/projected/6f092e55-100b-4423-b95f-be552520d428-kube-api-access-5fqhl\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391531 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddg27\" (UniqueName: \"kubernetes.io/projected/cb500cea-25dc-43d9-af5b-76a809317a4e-kube-api-access-ddg27\") pod \"network-check-source-685456655f-56mr8\" (UID: \"cb500cea-25dc-43d9-af5b-76a809317a4e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8"
Sep 08 04:01:04.391618 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6931a442-2287-43dd-b61f-93ffa6e89c5b-config\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.391672 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f092e55-100b-4423-b95f-be552520d428-serving-cert\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f092e55-100b-4423-b95f-be552520d428-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.391728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmq5\" (UniqueName: \"kubernetes.io/projected/2e181c82-660a-47df-b138-a3b81c374946-kube-api-access-4qmq5\") pod \"volume-data-source-validator-695c895bd6-r2vsj\" (UID: \"2e181c82-660a-47df-b138-a3b81c374946\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"
Sep 08 04:01:04.391728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.391696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f092e55-100b-4423-b95f-be552520d428-tmp\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.392143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.392123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6931a442-2287-43dd-b61f-93ffa6e89c5b-trusted-ca\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.392280 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.392261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6931a442-2287-43dd-b61f-93ffa6e89c5b-config\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.392334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.392323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6f092e55-100b-4423-b95f-be552520d428-snapshots\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.392476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.392459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f092e55-100b-4423-b95f-be552520d428-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.394427 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.394397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6931a442-2287-43dd-b61f-93ffa6e89c5b-serving-cert\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.394427 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.394419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f092e55-100b-4423-b95f-be552520d428-serving-cert\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.400681 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.400660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddg27\" (UniqueName: \"kubernetes.io/projected/cb500cea-25dc-43d9-af5b-76a809317a4e-kube-api-access-ddg27\") pod \"network-check-source-685456655f-56mr8\" (UID: \"cb500cea-25dc-43d9-af5b-76a809317a4e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8"
Sep 08 04:01:04.400813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.400678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqhl\" (UniqueName: \"kubernetes.io/projected/6f092e55-100b-4423-b95f-be552520d428-kube-api-access-5fqhl\") pod \"insights-operator-76bcd6ccf5-jpbbn\" (UID: \"6f092e55-100b-4423-b95f-be552520d428\") " pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.400938 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.400921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94d4b\" (UniqueName: \"kubernetes.io/projected/6931a442-2287-43dd-b61f-93ffa6e89c5b-kube-api-access-94d4b\") pod \"console-operator-85b454874b-8rqfd\" (UID: \"6931a442-2287-43dd-b61f-93ffa6e89c5b\") " pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.400975 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.400919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmq5\" (UniqueName: \"kubernetes.io/projected/2e181c82-660a-47df-b138-a3b81c374946-kube-api-access-4qmq5\") pod \"volume-data-source-validator-695c895bd6-r2vsj\" (UID: \"2e181c82-660a-47df-b138-a3b81c374946\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"
Sep 08 04:01:04.427784 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.427764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"
Sep 08 04:01:04.445915 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.445894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8"
Sep 08 04:01:04.458103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.458085    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn"
Sep 08 04:01:04.520103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.520071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:04.715877 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.715820    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj"]
Sep 08 04:01:04.718853 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:04.718828    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e181c82_660a_47df_b138_a3b81c374946.slice/crio-c4544e38d46faedbbeb7909fea4dc6f1d6707a7b6b2c5d16ac312b748b54bdd1 WatchSource:0}: Error finding container c4544e38d46faedbbeb7909fea4dc6f1d6707a7b6b2c5d16ac312b748b54bdd1: Status 404 returned error can't find the container with id c4544e38d46faedbbeb7909fea4dc6f1d6707a7b6b2c5d16ac312b748b54bdd1
Sep 08 04:01:04.748849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.748801    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-56mr8"]
Sep 08 04:01:04.752569 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:04.752542    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb500cea_25dc_43d9_af5b_76a809317a4e.slice/crio-2d8b09f9acb41d9d8551a8d93ae0c7b40edccb061086d1ad37d3a220f3ec2d1f WatchSource:0}: Error finding container 2d8b09f9acb41d9d8551a8d93ae0c7b40edccb061086d1ad37d3a220f3ec2d1f: Status 404 returned error can't find the container with id 2d8b09f9acb41d9d8551a8d93ae0c7b40edccb061086d1ad37d3a220f3ec2d1f
Sep 08 04:01:04.770039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.770009    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-jpbbn"]
Sep 08 04:01:04.772087 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:04.772056    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f092e55_100b_4423_b95f_be552520d428.slice/crio-b65cc7fbaabb820da98438159836aa22bc4e267c3a40da737f039d598945e431 WatchSource:0}: Error finding container b65cc7fbaabb820da98438159836aa22bc4e267c3a40da737f039d598945e431: Status 404 returned error can't find the container with id b65cc7fbaabb820da98438159836aa22bc4e267c3a40da737f039d598945e431
Sep 08 04:01:04.835814 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:04.835789    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8rqfd"]
Sep 08 04:01:04.839431 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:04.839408    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6931a442_2287_43dd_b61f_93ffa6e89c5b.slice/crio-62513c69da76f5cfb83432362c7e24ee5ad13a5c235616464a7936a3ec29f887 WatchSource:0}: Error finding container 62513c69da76f5cfb83432362c7e24ee5ad13a5c235616464a7936a3ec29f887: Status 404 returned error can't find the container with id 62513c69da76f5cfb83432362c7e24ee5ad13a5c235616464a7936a3ec29f887
Sep 08 04:01:05.325879 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:05.325843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn" event={"ID":"6f092e55-100b-4423-b95f-be552520d428","Type":"ContainerStarted","Data":"b65cc7fbaabb820da98438159836aa22bc4e267c3a40da737f039d598945e431"}
Sep 08 04:01:05.327292 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:05.327261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj" event={"ID":"2e181c82-660a-47df-b138-a3b81c374946","Type":"ContainerStarted","Data":"c4544e38d46faedbbeb7909fea4dc6f1d6707a7b6b2c5d16ac312b748b54bdd1"}
Sep 08 04:01:05.328685 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:05.328659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" event={"ID":"6931a442-2287-43dd-b61f-93ffa6e89c5b","Type":"ContainerStarted","Data":"62513c69da76f5cfb83432362c7e24ee5ad13a5c235616464a7936a3ec29f887"}
Sep 08 04:01:05.329961 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:05.329938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8" event={"ID":"cb500cea-25dc-43d9-af5b-76a809317a4e","Type":"ContainerStarted","Data":"2d8b09f9acb41d9d8551a8d93ae0c7b40edccb061086d1ad37d3a220f3ec2d1f"}
Sep 08 04:01:09.321704 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.321650    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"]
Sep 08 04:01:09.323652 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.323636    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"
Sep 08 04:01:09.326119 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.326097    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jrsff\""
Sep 08 04:01:09.326199 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.326108    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 04:01:09.327001 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.326984    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 04:01:09.337425 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.337406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"]
Sep 08 04:01:09.342398 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.342382    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/0.log"
Sep 08 04:01:09.342491 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.342416    2583 generic.go:356] "Generic (PLEG): container finished" podID="6931a442-2287-43dd-b61f-93ffa6e89c5b" containerID="89b4efc1eb1c8c6cbf0583ded77fd8f9b368bcefaa684aee2812551c4d8c0871" exitCode=255
Sep 08 04:01:09.342552 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.342496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" event={"ID":"6931a442-2287-43dd-b61f-93ffa6e89c5b","Type":"ContainerDied","Data":"89b4efc1eb1c8c6cbf0583ded77fd8f9b368bcefaa684aee2812551c4d8c0871"}
Sep 08 04:01:09.342669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.342657    2583 scope.go:117] "RemoveContainer" containerID="89b4efc1eb1c8c6cbf0583ded77fd8f9b368bcefaa684aee2812551c4d8c0871"
Sep 08 04:01:09.347680 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.347658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn" event={"ID":"6f092e55-100b-4423-b95f-be552520d428","Type":"ContainerStarted","Data":"29fb1923d244dbc3f8ddcc87597e2e70750031604b03445f572c842b58b6cdfa"}
Sep 08 04:01:09.349613 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.349590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj" event={"ID":"2e181c82-660a-47df-b138-a3b81c374946","Type":"ContainerStarted","Data":"9d03637f6e3c113f224002a7f60674445ff3325cbf8dd9758b9d3c05a87ad1e8"}
Sep 08 04:01:09.383132 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.383084    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn" podStartSLOduration=1.100046834 podStartE2EDuration="5.383064566s" totalImagesPullingTime="4.283017732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:09.382424006 +0000 UTC m=+49.740903566" watchObservedRunningTime="2026-09-08 04:01:09.383064566 +0000 UTC m=+49.741544128"
Sep 08 04:01:09.396392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.396346    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2vsj" podStartSLOduration=1.061370646 podStartE2EDuration="5.396329307s" totalImagesPullingTime="4.334958661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:09.396025831 +0000 UTC m=+49.754505392" watchObservedRunningTime="2026-09-08 04:01:09.396329307 +0000 UTC m=+49.754808868"
Sep 08 04:01:09.432337 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.432311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4bz\" (UniqueName: \"kubernetes.io/projected/52c19623-5420-4d23-bf2f-64c8df25d568-kube-api-access-jt4bz\") pod \"migrator-fdbb8fdff-hbj79\" (UID: \"52c19623-5420-4d23-bf2f-64c8df25d568\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"
Sep 08 04:01:09.533173 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.533141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4bz\" (UniqueName: \"kubernetes.io/projected/52c19623-5420-4d23-bf2f-64c8df25d568-kube-api-access-jt4bz\") pod \"migrator-fdbb8fdff-hbj79\" (UID: \"52c19623-5420-4d23-bf2f-64c8df25d568\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"
Sep 08 04:01:09.541826 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.541803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4bz\" (UniqueName: \"kubernetes.io/projected/52c19623-5420-4d23-bf2f-64c8df25d568-kube-api-access-jt4bz\") pod \"migrator-fdbb8fdff-hbj79\" (UID: \"52c19623-5420-4d23-bf2f-64c8df25d568\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"
Sep 08 04:01:09.637982 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.637933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"
Sep 08 04:01:09.910859 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:09.910645    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79"]
Sep 08 04:01:09.913063 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:09.913037    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c19623_5420_4d23_bf2f_64c8df25d568.slice/crio-2d7aa5243be2cc3ff811931835bd952a98d061aded68a86e4c7e4d2178a45258 WatchSource:0}: Error finding container 2d7aa5243be2cc3ff811931835bd952a98d061aded68a86e4c7e4d2178a45258: Status 404 returned error can't find the container with id 2d7aa5243be2cc3ff811931835bd952a98d061aded68a86e4c7e4d2178a45258
Sep 08 04:01:10.354569 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.354538    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/1.log"
Sep 08 04:01:10.355023 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.355005    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/0.log"
Sep 08 04:01:10.355094 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.355050    2583 generic.go:356] "Generic (PLEG): container finished" podID="6931a442-2287-43dd-b61f-93ffa6e89c5b" containerID="ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba" exitCode=255
Sep 08 04:01:10.355167 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.355140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" event={"ID":"6931a442-2287-43dd-b61f-93ffa6e89c5b","Type":"ContainerDied","Data":"ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba"}
Sep 08 04:01:10.355220 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.355203    2583 scope.go:117] "RemoveContainer" containerID="89b4efc1eb1c8c6cbf0583ded77fd8f9b368bcefaa684aee2812551c4d8c0871"
Sep 08 04:01:10.355371 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.355345    2583 scope.go:117] "RemoveContainer" containerID="ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba"
Sep 08 04:01:10.355584 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:10.355559    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8rqfd_openshift-console-operator(6931a442-2287-43dd-b61f-93ffa6e89c5b)\"" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" podUID="6931a442-2287-43dd-b61f-93ffa6e89c5b"
Sep 08 04:01:10.357010 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.356985    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8" event={"ID":"cb500cea-25dc-43d9-af5b-76a809317a4e","Type":"ContainerStarted","Data":"b9fde57c99e48861e625bb793c561f6e5943c99d3fa4b3281b039262ed0aa1f1"}
Sep 08 04:01:10.359516 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.359420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79" event={"ID":"52c19623-5420-4d23-bf2f-64c8df25d568","Type":"ContainerStarted","Data":"2d7aa5243be2cc3ff811931835bd952a98d061aded68a86e4c7e4d2178a45258"}
Sep 08 04:01:10.403300 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.403249    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-56mr8" podStartSLOduration=1.924680975 podStartE2EDuration="6.40323234s" totalImagesPullingTime="4.478551365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:10.40279554 +0000 UTC m=+50.761275101" watchObservedRunningTime="2026-09-08 04:01:10.40323234 +0000 UTC m=+50.761711901"
Sep 08 04:01:10.743114 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.743037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:10.743262 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:10.743128    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 04:01:10.743262 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:10.743139    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b748b67-h2bdx: secret "image-registry-tls" not found
Sep 08 04:01:10.743262 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:10.743181    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls podName:73078221-e155-4445-94e0-533af6d4b81a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.743169351 +0000 UTC m=+67.101648902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls") pod "image-registry-5f6b748b67-h2bdx" (UID: "73078221-e155-4445-94e0-533af6d4b81a") : secret "image-registry-tls" not found
Sep 08 04:01:10.844186 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:10.844159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:10.844339 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:10.844256    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 04:01:10.844339 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:10.844299    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls podName:4a6e4a1f-44cd-45a0-ba63-82faba49d733 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.844288685 +0000 UTC m=+67.202768227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls") pod "dns-default-m5h4q" (UID: "4a6e4a1f-44cd-45a0-ba63-82faba49d733") : secret "dns-default-metrics-tls" not found
Sep 08 04:01:11.041541 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.041509    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8xb48"]
Sep 08 04:01:11.043988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.043968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.046388 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.046367    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 04:01:11.046670 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.046649    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 04:01:11.046796 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.046684    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7wgqk\""
Sep 08 04:01:11.071677 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.071655    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8xb48"]
Sep 08 04:01:11.146066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.146040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb3e4f96-a892-4829-98c3-9284063b83fa-crio-socket\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.146160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.146079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.146160 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.146129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.146235 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.146186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e4f96-a892-4829-98c3-9284063b83fa-data-volume\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.146267 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.146238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwchb\" (UniqueName: \"kubernetes.io/projected/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-api-access-lwchb\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247345 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb3e4f96-a892-4829-98c3-9284063b83fa-crio-socket\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247454 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247454 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247454 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e4f96-a892-4829-98c3-9284063b83fa-data-volume\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247600 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.247455    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 04:01:11.247600 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.247512    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:11.247600 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.247521    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.74750177 +0000 UTC m=+52.105981326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : configmap "kube-rbac-proxy" not found
Sep 08 04:01:11.247600 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb3e4f96-a892-4829-98c3-9284063b83fa-crio-socket\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247600 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.247584    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.747566506 +0000 UTC m=+52.106046044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:11.247600 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwchb\" (UniqueName: \"kubernetes.io/projected/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-api-access-lwchb\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.247820 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.247700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb3e4f96-a892-4829-98c3-9284063b83fa-data-volume\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.263377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.263355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwchb\" (UniqueName: \"kubernetes.io/projected/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-api-access-lwchb\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.363847 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.363794    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/1.log"
Sep 08 04:01:11.364271 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.364253    2583 scope.go:117] "RemoveContainer" containerID="ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba"
Sep 08 04:01:11.364458 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.364424    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8rqfd_openshift-console-operator(6931a442-2287-43dd-b61f-93ffa6e89c5b)\"" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" podUID="6931a442-2287-43dd-b61f-93ffa6e89c5b"
Sep 08 04:01:11.696510 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.696445    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-q6rq6"]
Sep 08 04:01:11.700507 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.700165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.702909 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.702887    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hk7k6\""
Sep 08 04:01:11.703061 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.702912    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 04:01:11.703122 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.703103    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 04:01:11.703174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.702912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 04:01:11.703647 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.703627    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 04:01:11.719640 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.719617    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-q6rq6"]
Sep 08 04:01:11.751621 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.751598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.751727 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.751662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:11.751727 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.751713    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 04:01:11.751857 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.751799    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:12.75178163 +0000 UTC m=+53.110261178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : configmap "kube-rbac-proxy" not found
Sep 08 04:01:11.751857 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.751843    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:11.751965 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:11.751925    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:12.751906751 +0000 UTC m=+53.110386313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:11.852184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.852147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d5db688-1756-48e1-bd15-f2be3b8e9085-signing-key\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.852346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.852284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5db688-1756-48e1-bd15-f2be3b8e9085-signing-cabundle\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.852346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.852324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtb64\" (UniqueName: \"kubernetes.io/projected/8d5db688-1756-48e1-bd15-f2be3b8e9085-kube-api-access-wtb64\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.953328 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.953299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d5db688-1756-48e1-bd15-f2be3b8e9085-signing-key\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.953426 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.953412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5db688-1756-48e1-bd15-f2be3b8e9085-signing-cabundle\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.953493 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.953451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtb64\" (UniqueName: \"kubernetes.io/projected/8d5db688-1756-48e1-bd15-f2be3b8e9085-kube-api-access-wtb64\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.954103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.954083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5db688-1756-48e1-bd15-f2be3b8e9085-signing-cabundle\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.955935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.955886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d5db688-1756-48e1-bd15-f2be3b8e9085-signing-key\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:11.961865 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:11.961842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtb64\" (UniqueName: \"kubernetes.io/projected/8d5db688-1756-48e1-bd15-f2be3b8e9085-kube-api-access-wtb64\") pod \"service-ca-75d99c484-q6rq6\" (UID: \"8d5db688-1756-48e1-bd15-f2be3b8e9085\") " pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:12.010449 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.010419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-q6rq6"
Sep 08 04:01:12.286313 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.286283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-q6rq6"]
Sep 08 04:01:12.289541 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:12.289515    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5db688_1756_48e1_bd15_f2be3b8e9085.slice/crio-e0d1ba877e2c98850e80aeed2c064989d170c90fce5cc6e2de93877c29df6c0b WatchSource:0}: Error finding container e0d1ba877e2c98850e80aeed2c064989d170c90fce5cc6e2de93877c29df6c0b: Status 404 returned error can't find the container with id e0d1ba877e2c98850e80aeed2c064989d170c90fce5cc6e2de93877c29df6c0b
Sep 08 04:01:12.369071 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.369040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79" event={"ID":"52c19623-5420-4d23-bf2f-64c8df25d568","Type":"ContainerStarted","Data":"75239e7765dd0070d15e1f486b0146130d9b576c9bb042896d8e1723d02a4ec1"}
Sep 08 04:01:12.369402 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.369095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79" event={"ID":"52c19623-5420-4d23-bf2f-64c8df25d568","Type":"ContainerStarted","Data":"ecf27d56130926eaa94c689ece3d1b14cdcb8fdba367195b6090b085a810885a"}
Sep 08 04:01:12.370222 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.370199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-q6rq6" event={"ID":"8d5db688-1756-48e1-bd15-f2be3b8e9085","Type":"ContainerStarted","Data":"e0d1ba877e2c98850e80aeed2c064989d170c90fce5cc6e2de93877c29df6c0b"}
Sep 08 04:01:12.391188 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.391138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-hbj79" podStartSLOduration=1.418293537 podStartE2EDuration="3.39112324s" totalImagesPullingTime="1.972829703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:12.390161134 +0000 UTC m=+52.748640693" watchObservedRunningTime="2026-09-08 04:01:12.39112324 +0000 UTC m=+52.749602837"
Sep 08 04:01:12.759419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.759392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:12.759522 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:12.759439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:12.759522 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:12.759509    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 04:01:12.759587 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:12.759531    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:12.759587 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:12.759574    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:14.759562066 +0000 UTC m=+55.118041605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:12.759587 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:12.759587    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:14.759580821 +0000 UTC m=+55.118060358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : configmap "kube-rbac-proxy" not found
Sep 08 04:01:13.388691 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:13.388661    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rjjg2_02b6f1af-7eab-47aa-a84d-66a045c2dbdf/dns-node-resolver/0.log"
Sep 08 04:01:14.520825 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.520796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:14.521254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.520840    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:14.521254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.521129    2583 scope.go:117] "RemoveContainer" containerID="ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba"
Sep 08 04:01:14.521330 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:14.521259    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8rqfd_openshift-console-operator(6931a442-2287-43dd-b61f-93ffa6e89c5b)\"" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" podUID="6931a442-2287-43dd-b61f-93ffa6e89c5b"
Sep 08 04:01:14.589954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.589929    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xjmjj_6af1c09d-0ba6-4998-891f-0c61ed0f0f35/node-ca/0.log"
Sep 08 04:01:14.773655 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.773593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:14.773655 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.773638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:14.773826 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:14.773698    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 04:01:14.773826 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:14.773755    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:14.773826 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:14.773764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:18.773735098 +0000 UTC m=+59.132214635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : configmap "kube-rbac-proxy" not found
Sep 08 04:01:14.773826 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:14.773791    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:18.773781058 +0000 UTC m=+59.132260595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:14.991274 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:14.991248    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gh8zx_bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0/serve-healthcheck-canary/0.log"
Sep 08 04:01:15.390137 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:15.390109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-hbj79_52c19623-5420-4d23-bf2f-64c8df25d568/migrator/0.log"
Sep 08 04:01:15.590703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:15.590670    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-hbj79_52c19623-5420-4d23-bf2f-64c8df25d568/graceful-termination/0.log"
Sep 08 04:01:16.383958 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:16.383928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-q6rq6" event={"ID":"8d5db688-1756-48e1-bd15-f2be3b8e9085","Type":"ContainerStarted","Data":"eff8d39eedbc0b0f55eb6b312b4c9e2e68932b941dd5cb94f331ac7b0e4894a5"}
Sep 08 04:01:16.398725 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:16.398681    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-q6rq6" podStartSLOduration=1.701036243 podStartE2EDuration="5.398665739s" totalImagesPullingTime="3.697629496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:16.397706782 +0000 UTC m=+56.756186343" watchObservedRunningTime="2026-09-08 04:01:16.398665739 +0000 UTC m=+56.757145299"
Sep 08 04:01:18.804476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:18.804443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:18.804844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:18.804488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:18.804844 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:18.804567    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 04:01:18.804844 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:18.804632    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.804615608 +0000 UTC m=+67.163095146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : configmap "kube-rbac-proxy" not found
Sep 08 04:01:18.804844 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:18.804570    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:18.804844 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:18.804702    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.804688129 +0000 UTC m=+67.163167668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:21.294619 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:21.294593    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp22z"
Sep 08 04:01:25.958920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:25.958877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:01:25.959369 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:25.958937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:01:25.961542 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:25.961524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 04:01:25.962282 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:25.962268    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 04:01:25.972261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:25.972229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0ab2194-4d54-4200-91fa-6d67c62cec94-original-pull-secret\") pod \"global-pull-secret-syncer-lrg6q\" (UID: \"b0ab2194-4d54-4200-91fa-6d67c62cec94\") " pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:01:25.972390 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:25.972327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d77fff3-3663-446a-af43-cfd608056894-metrics-certs\") pod \"network-metrics-daemon-kj9xx\" (UID: \"4d77fff3-3663-446a-af43-cfd608056894\") " pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:01:26.059809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.059758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:01:26.062224 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.062196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2c2\" (UniqueName: \"kubernetes.io/projected/2d20074d-f010-4796-ae28-de01ed79d714-kube-api-access-dz2c2\") pod \"network-check-target-bw44h\" (UID: \"2d20074d-f010-4796-ae28-de01ed79d714\") " pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:01:26.066310 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.066285    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-blqwr\""
Sep 08 04:01:26.071218 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.071203    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrg6q"
Sep 08 04:01:26.074714 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.074700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj9xx"
Sep 08 04:01:26.080761 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.080747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f9485\""
Sep 08 04:01:26.088837 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.088803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:01:26.155853 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.154024    2583 scope.go:117] "RemoveContainer" containerID="ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba"
Sep 08 04:01:26.349135 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.349106    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kj9xx"]
Sep 08 04:01:26.351848 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.351817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lrg6q"]
Sep 08 04:01:26.352775 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:26.352687    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d77fff3_3663_446a_af43_cfd608056894.slice/crio-9c8e0692e64ea18d2f908f63f318b90d44b4d4460d5d935d056a23af58420abf WatchSource:0}: Error finding container 9c8e0692e64ea18d2f908f63f318b90d44b4d4460d5d935d056a23af58420abf: Status 404 returned error can't find the container with id 9c8e0692e64ea18d2f908f63f318b90d44b4d4460d5d935d056a23af58420abf
Sep 08 04:01:26.354796 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:26.354769    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ab2194_4d54_4200_91fa_6d67c62cec94.slice/crio-75d52a5f405456fef543aa0a79efc591570e06709febb3a4920ef89d8ae7098e WatchSource:0}: Error finding container 75d52a5f405456fef543aa0a79efc591570e06709febb3a4920ef89d8ae7098e: Status 404 returned error can't find the container with id 75d52a5f405456fef543aa0a79efc591570e06709febb3a4920ef89d8ae7098e
Sep 08 04:01:26.375124 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.375105    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bw44h"]
Sep 08 04:01:26.377092 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:26.377068    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d20074d_f010_4796_ae28_de01ed79d714.slice/crio-0794f0cffab601f74504641585f617e1f1a0692b91667c4fab613ae40744d2e8 WatchSource:0}: Error finding container 0794f0cffab601f74504641585f617e1f1a0692b91667c4fab613ae40744d2e8: Status 404 returned error can't find the container with id 0794f0cffab601f74504641585f617e1f1a0692b91667c4fab613ae40744d2e8
Sep 08 04:01:26.418007 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.417982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bw44h" event={"ID":"2d20074d-f010-4796-ae28-de01ed79d714","Type":"ContainerStarted","Data":"0794f0cffab601f74504641585f617e1f1a0692b91667c4fab613ae40744d2e8"}
Sep 08 04:01:26.419059 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.419029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lrg6q" event={"ID":"b0ab2194-4d54-4200-91fa-6d67c62cec94","Type":"ContainerStarted","Data":"75d52a5f405456fef543aa0a79efc591570e06709febb3a4920ef89d8ae7098e"}
Sep 08 04:01:26.420159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.420135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kj9xx" event={"ID":"4d77fff3-3663-446a-af43-cfd608056894","Type":"ContainerStarted","Data":"9c8e0692e64ea18d2f908f63f318b90d44b4d4460d5d935d056a23af58420abf"}
Sep 08 04:01:26.421619 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.421601    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:01:26.422007 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.421992    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/1.log"
Sep 08 04:01:26.422075 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.422024    2583 generic.go:356] "Generic (PLEG): container finished" podID="6931a442-2287-43dd-b61f-93ffa6e89c5b" containerID="63b79a2961a80780febc85866a0b8ae929dbc15c5f3f30a3b127fa84bcc4bb78" exitCode=255
Sep 08 04:01:26.422075 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.422063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" event={"ID":"6931a442-2287-43dd-b61f-93ffa6e89c5b","Type":"ContainerDied","Data":"63b79a2961a80780febc85866a0b8ae929dbc15c5f3f30a3b127fa84bcc4bb78"}
Sep 08 04:01:26.422139 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.422098    2583 scope.go:117] "RemoveContainer" containerID="ba6e97a686cff63fb2c1f0d8808a4fa53a2f2b0d4fc50e97572264103371bbba"
Sep 08 04:01:26.422404 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.422386    2583 scope.go:117] "RemoveContainer" containerID="63b79a2961a80780febc85866a0b8ae929dbc15c5f3f30a3b127fa84bcc4bb78"
Sep 08 04:01:26.422575 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:26.422555    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8rqfd_openshift-console-operator(6931a442-2287-43dd-b61f-93ffa6e89c5b)\"" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" podUID="6931a442-2287-43dd-b61f-93ffa6e89c5b"
Sep 08 04:01:26.766373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.766339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:26.768673 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.768651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"image-registry-5f6b748b67-h2bdx\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") " pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:26.866947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.866921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:26.867054 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.866954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:26.867054 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.866973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:26.867158 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:26.867082    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:26.867158 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:26.867130    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls podName:bb3e4f96-a892-4829-98c3-9284063b83fa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:42.867116852 +0000 UTC m=+83.225596390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xb48" (UID: "bb3e4f96-a892-4829-98c3-9284063b83fa") : secret "insights-runtime-extractor-tls" not found
Sep 08 04:01:26.867560 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.867537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb3e4f96-a892-4829-98c3-9284063b83fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:26.869118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:26.869097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a6e4a1f-44cd-45a0-ba63-82faba49d733-metrics-tls\") pod \"dns-default-m5h4q\" (UID: \"4a6e4a1f-44cd-45a0-ba63-82faba49d733\") " pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:27.058254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.058207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qx7pf\""
Sep 08 04:01:27.067094 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.067076    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:27.084959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.084932    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26sdq\""
Sep 08 04:01:27.092990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.092968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:27.366914 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.366749    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6b748b67-h2bdx"]
Sep 08 04:01:27.369251 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:27.369223    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73078221_e155_4445_94e0_533af6d4b81a.slice/crio-58c1c25a608225d7403053aa28e734d640eaad5d48bd7a49423bf638decde228 WatchSource:0}: Error finding container 58c1c25a608225d7403053aa28e734d640eaad5d48bd7a49423bf638decde228: Status 404 returned error can't find the container with id 58c1c25a608225d7403053aa28e734d640eaad5d48bd7a49423bf638decde228
Sep 08 04:01:27.389492 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.389468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5h4q"]
Sep 08 04:01:27.400645 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:27.400622    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6e4a1f_44cd_45a0_ba63_82faba49d733.slice/crio-43c1aa333ebf0a4053c54387b48c71fff4a7d9eca206dffa38c07a825fcdaa1a WatchSource:0}: Error finding container 43c1aa333ebf0a4053c54387b48c71fff4a7d9eca206dffa38c07a825fcdaa1a: Status 404 returned error can't find the container with id 43c1aa333ebf0a4053c54387b48c71fff4a7d9eca206dffa38c07a825fcdaa1a
Sep 08 04:01:27.426965 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.426937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bw44h" event={"ID":"2d20074d-f010-4796-ae28-de01ed79d714","Type":"ContainerStarted","Data":"0014ff1f4f62c69526d50675b8dc92d5633aa29b082d9370708e036333c917e1"}
Sep 08 04:01:27.427053 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.427041    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:01:27.428486 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.428467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" event={"ID":"73078221-e155-4445-94e0-533af6d4b81a","Type":"ContainerStarted","Data":"adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978"}
Sep 08 04:01:27.428588 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.428503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" event={"ID":"73078221-e155-4445-94e0-533af6d4b81a","Type":"ContainerStarted","Data":"58c1c25a608225d7403053aa28e734d640eaad5d48bd7a49423bf638decde228"}
Sep 08 04:01:27.428588 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.428562    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:27.429604 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.429550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5h4q" event={"ID":"4a6e4a1f-44cd-45a0-ba63-82faba49d733","Type":"ContainerStarted","Data":"43c1aa333ebf0a4053c54387b48c71fff4a7d9eca206dffa38c07a825fcdaa1a"}
Sep 08 04:01:27.431153 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.431137    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:01:27.443001 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.442957    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bw44h" podStartSLOduration=66.442945234 podStartE2EDuration="1m6.442945234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:27.442030141 +0000 UTC m=+67.800509701" watchObservedRunningTime="2026-09-08 04:01:27.442945234 +0000 UTC m=+67.801424793"
Sep 08 04:01:27.460626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:27.460591    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" podStartSLOduration=66.460579727 podStartE2EDuration="1m6.460579727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:27.45981329 +0000 UTC m=+67.818292849" watchObservedRunningTime="2026-09-08 04:01:27.460579727 +0000 UTC m=+67.819059280"
Sep 08 04:01:29.440133 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:29.440100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kj9xx" event={"ID":"4d77fff3-3663-446a-af43-cfd608056894","Type":"ContainerStarted","Data":"7001d04d1337358fc548db925115601dabd00a059a4de515874175b488748c6d"}
Sep 08 04:01:29.440556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:29.440158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kj9xx" event={"ID":"4d77fff3-3663-446a-af43-cfd608056894","Type":"ContainerStarted","Data":"aa2c2452354286039145c32783241f4226e9f511e02db5695d6c5ec4af05ebba"}
Sep 08 04:01:29.458645 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:29.458594    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kj9xx" podStartSLOduration=65.922243732 podStartE2EDuration="1m8.45857314s" totalImagesPullingTime="2.536329408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:29.458334459 +0000 UTC m=+69.816814021" watchObservedRunningTime="2026-09-08 04:01:29.45857314 +0000 UTC m=+69.817052701"
Sep 08 04:01:31.452932 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:31.452893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5h4q" event={"ID":"4a6e4a1f-44cd-45a0-ba63-82faba49d733","Type":"ContainerStarted","Data":"c56014b66df0f7d3a9493e0bd5d29b497fa9ece8626af7aa5abe8c79d1325d7f"}
Sep 08 04:01:31.453366 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:31.452976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5h4q" event={"ID":"4a6e4a1f-44cd-45a0-ba63-82faba49d733","Type":"ContainerStarted","Data":"142a87ad0669fd5fc3402e0f4200a1e372c693537e4064cc095910719b048087"}
Sep 08 04:01:31.453366 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:31.453027    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:31.471557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:31.471503    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m5h4q" podStartSLOduration=34.02210434 podStartE2EDuration="37.47148577s" totalImagesPullingTime="3.44938143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:31.470018888 +0000 UTC m=+71.828498450" watchObservedRunningTime="2026-09-08 04:01:31.47148577 +0000 UTC m=+71.829965331"
Sep 08 04:01:33.461440 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:33.461409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lrg6q" event={"ID":"b0ab2194-4d54-4200-91fa-6d67c62cec94","Type":"ContainerStarted","Data":"bb1b19b8b3ac367bc4f34ba21d11aaa72741ff8a1f9dc929fa6da904d99f5837"}
Sep 08 04:01:33.475999 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:33.475952    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lrg6q" podStartSLOduration=66.178267465 podStartE2EDuration="1m12.475936692s" totalImagesPullingTime="6.297669227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:33.475355802 +0000 UTC m=+73.833835364" watchObservedRunningTime="2026-09-08 04:01:33.475936692 +0000 UTC m=+73.834416252"
Sep 08 04:01:34.521264 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:34.521236    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:34.521599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:34.521281    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:34.521599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:34.521520    2583 scope.go:117] "RemoveContainer" containerID="63b79a2961a80780febc85866a0b8ae929dbc15c5f3f30a3b127fa84bcc4bb78"
Sep 08 04:01:34.521668 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:01:34.521641    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8rqfd_openshift-console-operator(6931a442-2287-43dd-b61f-93ffa6e89c5b)\"" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" podUID="6931a442-2287-43dd-b61f-93ffa6e89c5b"
Sep 08 04:01:35.799902 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.799867    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"]
Sep 08 04:01:35.802584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.802562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:35.805138 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.805117    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 04:01:35.805240 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.805190    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 04:01:35.806131 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.806109    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8ssjb\""
Sep 08 04:01:35.806215 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.806120    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 04:01:35.806297 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.806212    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 04:01:35.814081 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.814060    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"]
Sep 08 04:01:35.816050 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.816036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"]
Sep 08 04:01:35.816128 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.816116    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.819155 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.819136    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 04:01:35.819155 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.819143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 04:01:35.819290 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.819262    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 04:01:35.819386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.819370    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 04:01:35.827907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.827882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fac431f4-861a-4331-a003-2b380aab0121-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cc6684bb8-cghmd\" (UID: \"fac431f4-861a-4331-a003-2b380aab0121\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:35.828009 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.827928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44g6\" (UniqueName: \"kubernetes.io/projected/fac431f4-861a-4331-a003-2b380aab0121-kube-api-access-c44g6\") pod \"managed-serviceaccount-addon-agent-6cc6684bb8-cghmd\" (UID: \"fac431f4-861a-4331-a003-2b380aab0121\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:35.832511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.832492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"]
Sep 08 04:01:35.928654 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-ca\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.928788 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.928788 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.928788 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nkl\" (UniqueName: \"kubernetes.io/projected/dbf40157-0025-4105-b364-7b6f4da35709-kube-api-access-68nkl\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.928907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf40157-0025-4105-b364-7b6f4da35709-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.928907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fac431f4-861a-4331-a003-2b380aab0121-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cc6684bb8-cghmd\" (UID: \"fac431f4-861a-4331-a003-2b380aab0121\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:35.928907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-hub\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:35.928907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.928896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44g6\" (UniqueName: \"kubernetes.io/projected/fac431f4-861a-4331-a003-2b380aab0121-kube-api-access-c44g6\") pod \"managed-serviceaccount-addon-agent-6cc6684bb8-cghmd\" (UID: \"fac431f4-861a-4331-a003-2b380aab0121\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:35.931167 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.931144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fac431f4-861a-4331-a003-2b380aab0121-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cc6684bb8-cghmd\" (UID: \"fac431f4-861a-4331-a003-2b380aab0121\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:35.943683 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:35.943660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44g6\" (UniqueName: \"kubernetes.io/projected/fac431f4-861a-4331-a003-2b380aab0121-kube-api-access-c44g6\") pod \"managed-serviceaccount-addon-agent-6cc6684bb8-cghmd\" (UID: \"fac431f4-861a-4331-a003-2b380aab0121\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:36.029154 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.029129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-hub\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.029261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.029159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-ca\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.029261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.029180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.029261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.029224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.029261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.029250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nkl\" (UniqueName: \"kubernetes.io/projected/dbf40157-0025-4105-b364-7b6f4da35709-kube-api-access-68nkl\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.029469 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.029292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf40157-0025-4105-b364-7b6f4da35709-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.030114 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.030085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf40157-0025-4105-b364-7b6f4da35709-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.031540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.031516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.031624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.031559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-hub\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.031624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.031618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.031757 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.031715    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dbf40157-0025-4105-b364-7b6f4da35709-ca\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.042165 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.042142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nkl\" (UniqueName: \"kubernetes.io/projected/dbf40157-0025-4105-b364-7b6f4da35709-kube-api-access-68nkl\") pod \"cluster-proxy-proxy-agent-5c668689b5-g7hhl\" (UID: \"dbf40157-0025-4105-b364-7b6f4da35709\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.124614 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.124565    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"
Sep 08 04:01:36.134787 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.134765    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"
Sep 08 04:01:36.404915 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.404838    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd"]
Sep 08 04:01:36.407906 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:36.407866    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfac431f4_861a_4331_a003_2b380aab0121.slice/crio-82a919e002e370e61b0a17eb6857ff24668c45b45b24870d4ae229860d60eca5 WatchSource:0}: Error finding container 82a919e002e370e61b0a17eb6857ff24668c45b45b24870d4ae229860d60eca5: Status 404 returned error can't find the container with id 82a919e002e370e61b0a17eb6857ff24668c45b45b24870d4ae229860d60eca5
Sep 08 04:01:36.408651 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.408538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl"]
Sep 08 04:01:36.411380 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:36.411348    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbf40157_0025_4105_b364_7b6f4da35709.slice/crio-57b4e4a8238aadad92b2df562fdb64c64129b9d4114621e18b98a8d862d037e0 WatchSource:0}: Error finding container 57b4e4a8238aadad92b2df562fdb64c64129b9d4114621e18b98a8d862d037e0: Status 404 returned error can't find the container with id 57b4e4a8238aadad92b2df562fdb64c64129b9d4114621e18b98a8d862d037e0
Sep 08 04:01:36.471544 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.471521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl" event={"ID":"dbf40157-0025-4105-b364-7b6f4da35709","Type":"ContainerStarted","Data":"57b4e4a8238aadad92b2df562fdb64c64129b9d4114621e18b98a8d862d037e0"}
Sep 08 04:01:36.472694 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:36.472676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd" event={"ID":"fac431f4-861a-4331-a003-2b380aab0121","Type":"ContainerStarted","Data":"82a919e002e370e61b0a17eb6857ff24668c45b45b24870d4ae229860d60eca5"}
Sep 08 04:01:41.459089 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:41.459062    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m5h4q"
Sep 08 04:01:41.495286 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:41.495214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd" event={"ID":"fac431f4-861a-4331-a003-2b380aab0121","Type":"ContainerStarted","Data":"39998c9470c01b9172f289e8745cb3b6c482bf50a0f81ac47a90194402936122"}
Sep 08 04:01:41.497693 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:41.497656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl" event={"ID":"dbf40157-0025-4105-b364-7b6f4da35709","Type":"ContainerStarted","Data":"08e280bf585f963330d235eefbf40b2c4a3984742f8f7539542db7e4f8e82a7f"}
Sep 08 04:01:41.512551 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:41.512503    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cc6684bb8-cghmd" podStartSLOduration=1.592363612 podStartE2EDuration="6.512485227s" totalImagesPullingTime="4.920121615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:41.511770806 +0000 UTC m=+81.870250366" watchObservedRunningTime="2026-09-08 04:01:41.512485227 +0000 UTC m=+81.870964789"
Sep 08 04:01:42.502593 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:42.502558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl" event={"ID":"dbf40157-0025-4105-b364-7b6f4da35709","Type":"ContainerStarted","Data":"09bdcff2c9d2d37127c07c058d142938f83651c20999c1f2b0acbeb320f11f00"}
Sep 08 04:01:42.502945 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:42.502603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl" event={"ID":"dbf40157-0025-4105-b364-7b6f4da35709","Type":"ContainerStarted","Data":"dbd7ce256280ec5bed6195340db0a4ae8f18ee7b2bc8b8cc52bebe6f1b9284f3"}
Sep 08 04:01:42.522906 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:42.522862    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c668689b5-g7hhl" podStartSLOduration=2.587766053 podStartE2EDuration="7.522846642s" totalImagesPullingTime="4.935080589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:42.522098919 +0000 UTC m=+82.880578479" watchObservedRunningTime="2026-09-08 04:01:42.522846642 +0000 UTC m=+82.881326229"
Sep 08 04:01:42.884813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:42.884663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:42.886858 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:42.886829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb3e4f96-a892-4829-98c3-9284063b83fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xb48\" (UID: \"bb3e4f96-a892-4829-98c3-9284063b83fa\") " pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:43.156237 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:43.156174    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7wgqk\""
Sep 08 04:01:43.163476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:43.163456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8xb48"
Sep 08 04:01:43.447965 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:43.447943    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8xb48"]
Sep 08 04:01:43.450294 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:43.450271    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3e4f96_a892_4829_98c3_9284063b83fa.slice/crio-e455e426453eeb11136b6f5bc582b1d5ff4b4440583b05bffc7eb6e0629a5e96 WatchSource:0}: Error finding container e455e426453eeb11136b6f5bc582b1d5ff4b4440583b05bffc7eb6e0629a5e96: Status 404 returned error can't find the container with id e455e426453eeb11136b6f5bc582b1d5ff4b4440583b05bffc7eb6e0629a5e96
Sep 08 04:01:43.507710 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:43.507682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xb48" event={"ID":"bb3e4f96-a892-4829-98c3-9284063b83fa","Type":"ContainerStarted","Data":"e455e426453eeb11136b6f5bc582b1d5ff4b4440583b05bffc7eb6e0629a5e96"}
Sep 08 04:01:44.512253 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:44.512222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xb48" event={"ID":"bb3e4f96-a892-4829-98c3-9284063b83fa","Type":"ContainerStarted","Data":"7f134f7e2d6802b744d0f2142c4458d615bedf6ac0b32b58274068fbba1a76b6"}
Sep 08 04:01:44.512523 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:44.512273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xb48" event={"ID":"bb3e4f96-a892-4829-98c3-9284063b83fa","Type":"ContainerStarted","Data":"8fc24bbb5deebe0349b53ccc9c1e249da54cda1af66c72fcbfd280d9641a9220"}
Sep 08 04:01:47.072325 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.072257    2583 patch_prober.go:28] interesting pod/image-registry-5f6b748b67-h2bdx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 04:01:47.072812 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.072329    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" podUID="73078221-e155-4445-94e0-533af6d4b81a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 04:01:47.154124 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.153998    2583 scope.go:117] "RemoveContainer" containerID="63b79a2961a80780febc85866a0b8ae929dbc15c5f3f30a3b127fa84bcc4bb78"
Sep 08 04:01:47.525321 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.525290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xb48" event={"ID":"bb3e4f96-a892-4829-98c3-9284063b83fa","Type":"ContainerStarted","Data":"7b92cb558da3638b13704e41d1c8aa33cdda56b2a6d319463f004398d24f205e"}
Sep 08 04:01:47.527233 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.527214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:01:47.527348 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.527301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" event={"ID":"6931a442-2287-43dd-b61f-93ffa6e89c5b","Type":"ContainerStarted","Data":"e29db58dbbba58d0c2f660c45f1902df193135d17ddbd2f56a2640d2ceb8c786"}
Sep 08 04:01:47.527572 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.527556    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:47.544806 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.544765    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8xb48" podStartSLOduration=32.950658994 podStartE2EDuration="36.54472877s" totalImagesPullingTime="3.594069776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:01:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:47.544153017 +0000 UTC m=+87.902632577" watchObservedRunningTime="2026-09-08 04:01:47.54472877 +0000 UTC m=+87.903208329"
Sep 08 04:01:47.562347 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:47.562310    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-8rqfd" podStartSLOduration=39.341243396 podStartE2EDuration="43.5622966s" totalImagesPullingTime="4.221053204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:47.561379509 +0000 UTC m=+87.919859068" watchObservedRunningTime="2026-09-08 04:01:47.5622966 +0000 UTC m=+87.920776159"
Sep 08 04:01:48.006700 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:48.006675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-8rqfd"
Sep 08 04:01:48.439001 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:48.438932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:01:53.292616 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.292586    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bqbgr"]
Sep 08 04:01:53.297877 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.297852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.300854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.300826    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 04:01:53.301375 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.301353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 04:01:53.301491 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.301416    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 04:01:53.302149 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.302129    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 04:01:53.302240 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.302168    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nrb2v\""
Sep 08 04:01:53.302310 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.302290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 04:01:53.302413 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.302396    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 04:01:53.355916 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.355891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-root\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.355931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-textfile\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.355957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-wtmp\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.355983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-sys\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.356040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.356092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.356124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-metrics-client-ca\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356292 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.356191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-tls\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.356292 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.356227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ln7z\" (UniqueName: \"kubernetes.io/projected/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-kube-api-access-6ln7z\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457398 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-tls\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ln7z\" (UniqueName: \"kubernetes.io/projected/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-kube-api-access-6ln7z\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-root\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-root\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457661 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-textfile\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457661 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-wtmp\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457661 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-sys\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-sys\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-wtmp\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-metrics-client-ca\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.457990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.457916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-textfile\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.458495 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.458422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-metrics-client-ca\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.458495 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.458436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.459883 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.459863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-tls\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.459984 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.459967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.470432 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.470408    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ln7z\" (UniqueName: \"kubernetes.io/projected/4292fef7-0a29-4cab-9bdf-a64b1d477cc6-kube-api-access-6ln7z\") pod \"node-exporter-bqbgr\" (UID: \"4292fef7-0a29-4cab-9bdf-a64b1d477cc6\") " pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.609096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:53.609048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bqbgr"
Sep 08 04:01:53.619697 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:01:53.619677    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4292fef7_0a29_4cab_9bdf_a64b1d477cc6.slice/crio-110668e498ff0e4205a2917dbca1bf4bd458db17c5b9ca4afee752526102015c WatchSource:0}: Error finding container 110668e498ff0e4205a2917dbca1bf4bd458db17c5b9ca4afee752526102015c: Status 404 returned error can't find the container with id 110668e498ff0e4205a2917dbca1bf4bd458db17c5b9ca4afee752526102015c
Sep 08 04:01:54.551642 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:54.551603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqbgr" event={"ID":"4292fef7-0a29-4cab-9bdf-a64b1d477cc6","Type":"ContainerStarted","Data":"110668e498ff0e4205a2917dbca1bf4bd458db17c5b9ca4afee752526102015c"}
Sep 08 04:01:55.556854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:55.556820    2583 generic.go:356] "Generic (PLEG): container finished" podID="4292fef7-0a29-4cab-9bdf-a64b1d477cc6" containerID="791e214a5d00c62cb2ce1d1fd710f9f331c5b3077d18caf2136171654b0d0dbd" exitCode=0
Sep 08 04:01:55.557219 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:55.556895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqbgr" event={"ID":"4292fef7-0a29-4cab-9bdf-a64b1d477cc6","Type":"ContainerDied","Data":"791e214a5d00c62cb2ce1d1fd710f9f331c5b3077d18caf2136171654b0d0dbd"}
Sep 08 04:01:56.562489 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:56.562450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqbgr" event={"ID":"4292fef7-0a29-4cab-9bdf-a64b1d477cc6","Type":"ContainerStarted","Data":"17827b0d53ab6783aaaaeab0b90c6e10a25909ed5d45eb0e45e55ab0839422be"}
Sep 08 04:01:56.562857 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:56.562501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqbgr" event={"ID":"4292fef7-0a29-4cab-9bdf-a64b1d477cc6","Type":"ContainerStarted","Data":"df5c3caaf64ab4a6744ed81638bff9e14326e0466ebf61882603f57059678f1c"}
Sep 08 04:01:56.584254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:56.584212    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bqbgr" podStartSLOduration=2.4942921829999998 podStartE2EDuration="3.584197123s" totalImagesPullingTime="1.031021497s" totalInitContainerRuntime="58.883443ms" isStatefulPod=true podCreationTimestamp="2026-09-08 04:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:56.58369653 +0000 UTC m=+96.942176087" watchObservedRunningTime="2026-09-08 04:01:56.584197123 +0000 UTC m=+96.942676682"
Sep 08 04:01:58.436836 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:58.436804    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bw44h"
Sep 08 04:01:59.930305 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:01:59.930269    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6b748b67-h2bdx"]
Sep 08 04:02:20.649163 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:20.649124    2583 generic.go:356] "Generic (PLEG): container finished" podID="6f092e55-100b-4423-b95f-be552520d428" containerID="29fb1923d244dbc3f8ddcc87597e2e70750031604b03445f572c842b58b6cdfa" exitCode=0
Sep 08 04:02:20.649556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:20.649198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn" event={"ID":"6f092e55-100b-4423-b95f-be552520d428","Type":"ContainerDied","Data":"29fb1923d244dbc3f8ddcc87597e2e70750031604b03445f572c842b58b6cdfa"}
Sep 08 04:02:20.649556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:20.649490    2583 scope.go:117] "RemoveContainer" containerID="29fb1923d244dbc3f8ddcc87597e2e70750031604b03445f572c842b58b6cdfa"
Sep 08 04:02:21.208605 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:21.208535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gh8zx_bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0/serve-healthcheck-canary/0.log"
Sep 08 04:02:21.656808 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:21.656768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-jpbbn" event={"ID":"6f092e55-100b-4423-b95f-be552520d428","Type":"ContainerStarted","Data":"daa1a7db4a8c57efac88e8e6a574e3da5d1c3e337d24e27a849541f25265b03b"}
Sep 08 04:02:24.949753 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:24.949698    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" podUID="73078221-e155-4445-94e0-533af6d4b81a" containerName="registry" containerID="cri-o://adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978" gracePeriod=30
Sep 08 04:02:25.188910 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.188888    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:02:25.285377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285353    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-registry-certificates\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-image-registry-private-configuration\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285427    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73078221-e155-4445-94e0-533af6d4b81a-ca-trust-extracted\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285457    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnrps\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-kube-api-access-jnrps\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285501    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-bound-sa-token\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285732 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285540    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-installation-pull-secrets\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.285860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:02:25.285975 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.285956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-trusted-ca\") pod \"73078221-e155-4445-94e0-533af6d4b81a\" (UID: \"73078221-e155-4445-94e0-533af6d4b81a\") "
Sep 08 04:02:25.286203 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.286174    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-registry-certificates\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.286692 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.286656    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:02:25.288570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.288529    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:02:25.288670 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.288583    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:02:25.288670 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.288583    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-kube-api-access-jnrps" (OuterVolumeSpecName: "kube-api-access-jnrps") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "kube-api-access-jnrps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:02:25.288670 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.288653    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:02:25.288850 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.288759    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:02:25.293979 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.293954    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73078221-e155-4445-94e0-533af6d4b81a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73078221-e155-4445-94e0-533af6d4b81a" (UID: "73078221-e155-4445-94e0-533af6d4b81a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:02:25.386715 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386693    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73078221-e155-4445-94e0-533af6d4b81a-ca-trust-extracted\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.386715 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386714    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnrps\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-kube-api-access-jnrps\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.386844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386723    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-registry-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.386844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386732    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73078221-e155-4445-94e0-533af6d4b81a-bound-sa-token\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.386844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386754    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-installation-pull-secrets\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.386844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386765    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73078221-e155-4445-94e0-533af6d4b81a-trusted-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.386844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.386774    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73078221-e155-4445-94e0-533af6d4b81a-image-registry-private-configuration\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:02:25.673446 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.673381    2583 generic.go:356] "Generic (PLEG): container finished" podID="73078221-e155-4445-94e0-533af6d4b81a" containerID="adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978" exitCode=0
Sep 08 04:02:25.673446 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.673416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" event={"ID":"73078221-e155-4445-94e0-533af6d4b81a","Type":"ContainerDied","Data":"adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978"}
Sep 08 04:02:25.673446 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.673426    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx"
Sep 08 04:02:25.673638 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.673465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b748b67-h2bdx" event={"ID":"73078221-e155-4445-94e0-533af6d4b81a","Type":"ContainerDied","Data":"58c1c25a608225d7403053aa28e734d640eaad5d48bd7a49423bf638decde228"}
Sep 08 04:02:25.673638 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.673493    2583 scope.go:117] "RemoveContainer" containerID="adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978"
Sep 08 04:02:25.681565 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.681549    2583 scope.go:117] "RemoveContainer" containerID="adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978"
Sep 08 04:02:25.681869 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:02:25.681851    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978\": container with ID starting with adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978 not found: ID does not exist" containerID="adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978"
Sep 08 04:02:25.681932 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.681877    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978"} err="failed to get container status \"adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978\": rpc error: code = NotFound desc = could not find container \"adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978\": container with ID starting with adeea95aebed776b7f2fb18a8f1ef34b5a294a8abc54958a9e2031941f2da978 not found: ID does not exist"
Sep 08 04:02:25.703695 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.703669    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6b748b67-h2bdx"]
Sep 08 04:02:25.711554 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:25.711536    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f6b748b67-h2bdx"]
Sep 08 04:02:26.158239 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:02:26.158213    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73078221-e155-4445-94e0-533af6d4b81a" path="/var/lib/kubelet/pods/73078221-e155-4445-94e0-533af6d4b81a/volumes"
Sep 08 04:05:20.069520 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:05:20.069493    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:05:20.073429 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:05:20.073407    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:05:20.080965 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:05:20.080946    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 04:06:36.984988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.984956    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"]
Sep 08 04:06:36.985424 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.985224    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73078221-e155-4445-94e0-533af6d4b81a" containerName="registry"
Sep 08 04:06:36.985424 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.985236    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="73078221-e155-4445-94e0-533af6d4b81a" containerName="registry"
Sep 08 04:06:36.985424 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.985290    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="73078221-e155-4445-94e0-533af6d4b81a" containerName="registry"
Sep 08 04:06:36.988053 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.988036    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:36.991151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.991128    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 04:06:36.991618 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.991601    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 04:06:36.991733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.991714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jkdxt\""
Sep 08 04:06:36.991905 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.991881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 04:06:36.992441 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:36.992427    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 04:06:37.005807 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.005781    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"]
Sep 08 04:06:37.124953 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.124928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aad681d-b91c-440c-85dc-4f326e591c93-tmp\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.125045 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.124955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aad681d-b91c-440c-85dc-4f326e591c93-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.125045 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.124998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9aad681d-b91c-440c-85dc-4f326e591c93-manifests\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.125114 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.125060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkn5\" (UniqueName: \"kubernetes.io/projected/9aad681d-b91c-440c-85dc-4f326e591c93-kube-api-access-ptkn5\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.125151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.125116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9aad681d-b91c-440c-85dc-4f326e591c93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.226343 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.226312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkn5\" (UniqueName: \"kubernetes.io/projected/9aad681d-b91c-440c-85dc-4f326e591c93-kube-api-access-ptkn5\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.226434 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.226347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9aad681d-b91c-440c-85dc-4f326e591c93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.226473 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.226451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aad681d-b91c-440c-85dc-4f326e591c93-tmp\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.226473 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.226469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aad681d-b91c-440c-85dc-4f326e591c93-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.226546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.226511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9aad681d-b91c-440c-85dc-4f326e591c93-manifests\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.226908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.226876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9aad681d-b91c-440c-85dc-4f326e591c93-manifests\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.228647 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.228618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aad681d-b91c-440c-85dc-4f326e591c93-tmp\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.228647 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.228640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9aad681d-b91c-440c-85dc-4f326e591c93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.228821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.228806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9aad681d-b91c-440c-85dc-4f326e591c93-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.238518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.238470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkn5\" (UniqueName: \"kubernetes.io/projected/9aad681d-b91c-440c-85dc-4f326e591c93-kube-api-access-ptkn5\") pod \"opendatahub-operator-controller-manager-5d5d59f488-5prhf\" (UID: \"9aad681d-b91c-440c-85dc-4f326e591c93\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.298046 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.298023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:37.571669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.571639    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"]
Sep 08 04:06:37.574591 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:06:37.574569    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aad681d_b91c_440c_85dc_4f326e591c93.slice/crio-827df4b56b6f70e55d7142fe3d7248336c7559fd72e857a926a162f31b562cb6 WatchSource:0}: Error finding container 827df4b56b6f70e55d7142fe3d7248336c7559fd72e857a926a162f31b562cb6: Status 404 returned error can't find the container with id 827df4b56b6f70e55d7142fe3d7248336c7559fd72e857a926a162f31b562cb6
Sep 08 04:06:37.576249 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:37.576229    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 04:06:38.512799 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:38.512761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf" event={"ID":"9aad681d-b91c-440c-85dc-4f326e591c93","Type":"ContainerStarted","Data":"827df4b56b6f70e55d7142fe3d7248336c7559fd72e857a926a162f31b562cb6"}
Sep 08 04:06:41.525712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:41.525671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf" event={"ID":"9aad681d-b91c-440c-85dc-4f326e591c93","Type":"ContainerStarted","Data":"892d0fff587c047a602593a4d1c8b1dfd3f8481a78ef2cb77fe12aa98ca9757e"}
Sep 08 04:06:42.529919 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:42.529888    2583 generic.go:356] "Generic (PLEG): container finished" podID="9aad681d-b91c-440c-85dc-4f326e591c93" containerID="892d0fff587c047a602593a4d1c8b1dfd3f8481a78ef2cb77fe12aa98ca9757e" exitCode=0
Sep 08 04:06:42.530265 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:42.529957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf" event={"ID":"9aad681d-b91c-440c-85dc-4f326e591c93","Type":"ContainerDied","Data":"892d0fff587c047a602593a4d1c8b1dfd3f8481a78ef2cb77fe12aa98ca9757e"}
Sep 08 04:06:43.536275 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:43.536244    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf" event={"ID":"9aad681d-b91c-440c-85dc-4f326e591c93","Type":"ContainerStarted","Data":"b558461948afb7f98535120cf75d0b34b4783e8337cd97ff6e403b6978d3f8de"}
Sep 08 04:06:43.536635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:43.536375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:43.562239 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:43.562196    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf" podStartSLOduration=3.2597875370000002 podStartE2EDuration="7.562181435s" totalImagesPullingTime="3.246248527s" totalInitContainerRuntime="1.056145371s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:06:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:06:43.560099548 +0000 UTC m=+383.918579111" watchObservedRunningTime="2026-09-08 04:06:43.562181435 +0000 UTC m=+383.920660995"
Sep 08 04:06:54.541299 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:54.541267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-5prhf"
Sep 08 04:06:55.167156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.167124    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"]
Sep 08 04:06:55.176531 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.176508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.179062 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.179014    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 04:06:55.179219 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.179182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 04:06:55.180149 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.180130    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 04:06:55.180260 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.180238    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 04:06:55.180342 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.180318    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-55m7s\""
Sep 08 04:06:55.180342 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.180341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 04:06:55.182248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.182227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"]
Sep 08 04:06:55.345837 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.345807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-metrics-cert\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.345988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.345847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-cert\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.345988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.345869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qks\" (UniqueName: \"kubernetes.io/projected/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-kube-api-access-26qks\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.345988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.345948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-manager-config\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.446783 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.446712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-cert\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.446783 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.446758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qks\" (UniqueName: \"kubernetes.io/projected/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-kube-api-access-26qks\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.446927 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.446789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-manager-config\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.446927 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.446820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-metrics-cert\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.447429 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.447410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-manager-config\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.449262 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.449239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-cert\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.449340 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.449239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-metrics-cert\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.458793 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.458772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qks\" (UniqueName: \"kubernetes.io/projected/afd765a2-c2c6-4919-bc0d-eff1b3e114ab-kube-api-access-26qks\") pod \"lws-controller-manager-58d7df9668-7jwnc\" (UID: \"afd765a2-c2c6-4919-bc0d-eff1b3e114ab\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.486803 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.486783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:55.758317 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:55.758232    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"]
Sep 08 04:06:55.760487 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:06:55.760461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd765a2_c2c6_4919_bc0d_eff1b3e114ab.slice/crio-39d845d067d89b6d20ba601f639017f098188c1a6f6e9c863f9bb515851ec8cd WatchSource:0}: Error finding container 39d845d067d89b6d20ba601f639017f098188c1a6f6e9c863f9bb515851ec8cd: Status 404 returned error can't find the container with id 39d845d067d89b6d20ba601f639017f098188c1a6f6e9c863f9bb515851ec8cd
Sep 08 04:06:56.581882 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.581845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc" event={"ID":"afd765a2-c2c6-4919-bc0d-eff1b3e114ab","Type":"ContainerStarted","Data":"39d845d067d89b6d20ba601f639017f098188c1a6f6e9c863f9bb515851ec8cd"}
Sep 08 04:06:56.754362 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.752425    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"]
Sep 08 04:06:56.756178 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.756149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.761363 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.761336    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 04:06:56.761896 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.761830    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 04:06:56.761896 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.761866    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z6qkw\""
Sep 08 04:06:56.762355 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.762327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 04:06:56.762550 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.762536    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 04:06:56.773159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.773136    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"]
Sep 08 04:06:56.857685 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.857489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tls-certs\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.857685 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.857529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tmp\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.857685 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.857613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hjm\" (UniqueName: \"kubernetes.io/projected/6dbf819d-bfb6-4491-80bd-60238fb4bf27-kube-api-access-m2hjm\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.959003 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.958567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tls-certs\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.959003 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.958608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tmp\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.959003 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.958652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hjm\" (UniqueName: \"kubernetes.io/projected/6dbf819d-bfb6-4491-80bd-60238fb4bf27-kube-api-access-m2hjm\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.959003 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:06:56.958673    2583 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 04:06:56.959003 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:06:56.958764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tls-certs podName:6dbf819d-bfb6-4491-80bd-60238fb4bf27 nodeName:}" failed. No retries permitted until 2026-09-08 04:06:57.458727417 +0000 UTC m=+397.817206975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tls-certs") pod "kube-auth-proxy-5859ff6677-lllzs" (UID: "6dbf819d-bfb6-4491-80bd-60238fb4bf27") : secret "kube-auth-proxy-tls" not found
Sep 08 04:06:56.961505 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.961470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tmp\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:56.971993 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:56.971969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hjm\" (UniqueName: \"kubernetes.io/projected/6dbf819d-bfb6-4491-80bd-60238fb4bf27-kube-api-access-m2hjm\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:57.463314 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:57.463278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tls-certs\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:57.465426 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:57.465406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbf819d-bfb6-4491-80bd-60238fb4bf27-tls-certs\") pod \"kube-auth-proxy-5859ff6677-lllzs\" (UID: \"6dbf819d-bfb6-4491-80bd-60238fb4bf27\") " pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:57.667159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:57.667110    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"
Sep 08 04:06:58.591698 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:58.591669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc" event={"ID":"afd765a2-c2c6-4919-bc0d-eff1b3e114ab","Type":"ContainerStarted","Data":"bb69e6c0ce84bb3636c4e783ab1c12c45786bec3e22849b1cd8b89a28621edbe"}
Sep 08 04:06:58.592058 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:58.591889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:06:58.610148 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:58.610102    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc" podStartSLOduration=0.863575702 podStartE2EDuration="3.610085497s" totalImagesPullingTime="2.746509795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:06:58.608283832 +0000 UTC m=+398.966763393" watchObservedRunningTime="2026-09-08 04:06:58.610085497 +0000 UTC m=+398.968565059"
Sep 08 04:06:58.705473 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:58.705450    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5859ff6677-lllzs"]
Sep 08 04:06:58.707105 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:06:58.707079    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbf819d_bfb6_4491_80bd_60238fb4bf27.slice/crio-a25f02bf5436948f20188bceed24d09df483b8a5c4809176f9ae354f913b3e60 WatchSource:0}: Error finding container a25f02bf5436948f20188bceed24d09df483b8a5c4809176f9ae354f913b3e60: Status 404 returned error can't find the container with id a25f02bf5436948f20188bceed24d09df483b8a5c4809176f9ae354f913b3e60
Sep 08 04:06:59.598341 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:06:59.598301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs" event={"ID":"6dbf819d-bfb6-4491-80bd-60238fb4bf27","Type":"ContainerStarted","Data":"a25f02bf5436948f20188bceed24d09df483b8a5c4809176f9ae354f913b3e60"}
Sep 08 04:07:01.607856 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:07:01.607767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs" event={"ID":"6dbf819d-bfb6-4491-80bd-60238fb4bf27","Type":"ContainerStarted","Data":"e46bcb41ea2855cf23df6cfaca347d57a476a9f23b098461202bd1015f5f2362"}
Sep 08 04:07:01.623567 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:07:01.623522    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5859ff6677-lllzs" podStartSLOduration=3.059990232 podStartE2EDuration="5.623508938s" totalImagesPullingTime="2.563518706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:01.622502978 +0000 UTC m=+401.980982538" watchObservedRunningTime="2026-09-08 04:07:01.623508938 +0000 UTC m=+401.981988507"
Sep 08 04:07:09.601235 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:07:09.601202    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-7jwnc"
Sep 08 04:08:38.651408 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.651372    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"]
Sep 08 04:08:38.653663 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.653641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.656098 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.656065    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 04:08:38.656098 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.656081    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 04:08:38.656272 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.656167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 04:08:38.656929 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.656914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d4vgw\""
Sep 08 04:08:38.656929 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.656923    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 04:08:38.663719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.663700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"]
Sep 08 04:08:38.743444 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.743424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a71f0a-2d06-4b91-ac86-44754b674b43-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.743532 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.743456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/84a71f0a-2d06-4b91-ac86-44754b674b43-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.743532 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.743500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8wc\" (UniqueName: \"kubernetes.io/projected/84a71f0a-2d06-4b91-ac86-44754b674b43-kube-api-access-2r8wc\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.844654 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.844633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/84a71f0a-2d06-4b91-ac86-44754b674b43-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.844729 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.844672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8wc\" (UniqueName: \"kubernetes.io/projected/84a71f0a-2d06-4b91-ac86-44754b674b43-kube-api-access-2r8wc\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.844804 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.844727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a71f0a-2d06-4b91-ac86-44754b674b43-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.844855 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:08:38.844840    2583 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 08 04:08:38.844927 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:08:38.844915    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a71f0a-2d06-4b91-ac86-44754b674b43-plugin-serving-cert podName:84a71f0a-2d06-4b91-ac86-44754b674b43 nodeName:}" failed. No retries permitted until 2026-09-08 04:08:39.344895042 +0000 UTC m=+499.703374595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/84a71f0a-2d06-4b91-ac86-44754b674b43-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-gkn4k" (UID: "84a71f0a-2d06-4b91-ac86-44754b674b43") : secret "plugin-serving-cert" not found
Sep 08 04:08:38.845330 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.845313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/84a71f0a-2d06-4b91-ac86-44754b674b43-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:38.852788 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:38.852767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8wc\" (UniqueName: \"kubernetes.io/projected/84a71f0a-2d06-4b91-ac86-44754b674b43-kube-api-access-2r8wc\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:39.347150 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:39.347126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a71f0a-2d06-4b91-ac86-44754b674b43-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:39.349182 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:39.349164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/84a71f0a-2d06-4b91-ac86-44754b674b43-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkn4k\" (UID: \"84a71f0a-2d06-4b91-ac86-44754b674b43\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:39.562678 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:39.562655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"
Sep 08 04:08:39.831435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:39.831405    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k"]
Sep 08 04:08:39.834785 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:08:39.834758    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a71f0a_2d06_4b91_ac86_44754b674b43.slice/crio-0b13a0f4106cd271629a9401a399a7f014471cdb27bbf5db0cbb2a5011bc700d WatchSource:0}: Error finding container 0b13a0f4106cd271629a9401a399a7f014471cdb27bbf5db0cbb2a5011bc700d: Status 404 returned error can't find the container with id 0b13a0f4106cd271629a9401a399a7f014471cdb27bbf5db0cbb2a5011bc700d
Sep 08 04:08:39.993208 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:39.993146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k" event={"ID":"84a71f0a-2d06-4b91-ac86-44754b674b43","Type":"ContainerStarted","Data":"0b13a0f4106cd271629a9401a399a7f014471cdb27bbf5db0cbb2a5011bc700d"}
Sep 08 04:08:47.026999 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:47.026967    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k" event={"ID":"84a71f0a-2d06-4b91-ac86-44754b674b43","Type":"ContainerStarted","Data":"0344c80307165095895437bb0c0fd18ee6fa10c90998c0af12751a054fb1d844"}
Sep 08 04:08:47.044232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:08:47.044186    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkn4k" podStartSLOduration=2.40063396 podStartE2EDuration="9.044170365s" totalImagesPullingTime="6.643536405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:08:47.042847356 +0000 UTC m=+507.401326917" watchObservedRunningTime="2026-09-08 04:08:47.044170365 +0000 UTC m=+507.402649925"
Sep 08 04:09:32.499132 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.499089    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:09:32.501413 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.501388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.503890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.503871    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 04:09:32.513323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.513302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:09:32.548576 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.548554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:09:32.607581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.607561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/957ca297-c9b6-4c20-8ed3-149731a44ee4-config-file\") pod \"limitador-limitador-85d9db4d5d-dccpl\" (UID: \"957ca297-c9b6-4c20-8ed3-149731a44ee4\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.607668 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.607603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j82z\" (UniqueName: \"kubernetes.io/projected/957ca297-c9b6-4c20-8ed3-149731a44ee4-kube-api-access-6j82z\") pod \"limitador-limitador-85d9db4d5d-dccpl\" (UID: \"957ca297-c9b6-4c20-8ed3-149731a44ee4\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.708894 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.708871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j82z\" (UniqueName: \"kubernetes.io/projected/957ca297-c9b6-4c20-8ed3-149731a44ee4-kube-api-access-6j82z\") pod \"limitador-limitador-85d9db4d5d-dccpl\" (UID: \"957ca297-c9b6-4c20-8ed3-149731a44ee4\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.708980 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.708924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/957ca297-c9b6-4c20-8ed3-149731a44ee4-config-file\") pod \"limitador-limitador-85d9db4d5d-dccpl\" (UID: \"957ca297-c9b6-4c20-8ed3-149731a44ee4\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.709463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.709443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/957ca297-c9b6-4c20-8ed3-149731a44ee4-config-file\") pod \"limitador-limitador-85d9db4d5d-dccpl\" (UID: \"957ca297-c9b6-4c20-8ed3-149731a44ee4\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.716410 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.716391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j82z\" (UniqueName: \"kubernetes.io/projected/957ca297-c9b6-4c20-8ed3-149731a44ee4-kube-api-access-6j82z\") pod \"limitador-limitador-85d9db4d5d-dccpl\" (UID: \"957ca297-c9b6-4c20-8ed3-149731a44ee4\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.810988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.810968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:32.862729 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.862701    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-sgfxt"]
Sep 08 04:09:32.865295 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.865273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:32.867526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.867498    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7kn75\""
Sep 08 04:09:32.874258 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:32.874216    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-sgfxt"]
Sep 08 04:09:33.011072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.011045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjd5q\" (UniqueName: \"kubernetes.io/projected/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59-kube-api-access-tjd5q\") pod \"authorino-b54df94b4-sgfxt\" (UID: \"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59\") " pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:33.017394 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.017372    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vkrsg"]
Sep 08 04:09:33.019182 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.019167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:09:33.028568 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.028550    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vkrsg"]
Sep 08 04:09:33.076807 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.076756    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:09:33.079586 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:09:33.079563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957ca297_c9b6_4c20_8ed3_149731a44ee4.slice/crio-86ad6edaaeb4c6d2983bf9680e9d02777b4f8c544575477860784192a8a23844 WatchSource:0}: Error finding container 86ad6edaaeb4c6d2983bf9680e9d02777b4f8c544575477860784192a8a23844: Status 404 returned error can't find the container with id 86ad6edaaeb4c6d2983bf9680e9d02777b4f8c544575477860784192a8a23844
Sep 08 04:09:33.112312 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.112288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjd5q\" (UniqueName: \"kubernetes.io/projected/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59-kube-api-access-tjd5q\") pod \"authorino-b54df94b4-sgfxt\" (UID: \"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59\") " pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:33.120885 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.120861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjd5q\" (UniqueName: \"kubernetes.io/projected/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59-kube-api-access-tjd5q\") pod \"authorino-b54df94b4-sgfxt\" (UID: \"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59\") " pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:33.178990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.178971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:33.203555 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.203534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl" event={"ID":"957ca297-c9b6-4c20-8ed3-149731a44ee4","Type":"ContainerStarted","Data":"86ad6edaaeb4c6d2983bf9680e9d02777b4f8c544575477860784192a8a23844"}
Sep 08 04:09:33.213448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.213410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcb7\" (UniqueName: \"kubernetes.io/projected/b7279f48-120e-42aa-80aa-259fc7f2ae5c-kube-api-access-nmcb7\") pod \"authorino-c548b5d7d-vkrsg\" (UID: \"b7279f48-120e-42aa-80aa-259fc7f2ae5c\") " pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:09:33.314840 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.314811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcb7\" (UniqueName: \"kubernetes.io/projected/b7279f48-120e-42aa-80aa-259fc7f2ae5c-kube-api-access-nmcb7\") pod \"authorino-c548b5d7d-vkrsg\" (UID: \"b7279f48-120e-42aa-80aa-259fc7f2ae5c\") " pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:09:33.322781 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.322762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcb7\" (UniqueName: \"kubernetes.io/projected/b7279f48-120e-42aa-80aa-259fc7f2ae5c-kube-api-access-nmcb7\") pod \"authorino-c548b5d7d-vkrsg\" (UID: \"b7279f48-120e-42aa-80aa-259fc7f2ae5c\") " pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:09:33.327510 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.327472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:09:33.441141 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.441118    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-sgfxt"]
Sep 08 04:09:33.443404 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:09:33.443377    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2af5f056_99c7_4a69_96fb_fbdb1ac0ac59.slice/crio-e7fec342d6be2413a873c38e5b896e70b21ebb3fa32197a10ebb49f59167087c WatchSource:0}: Error finding container e7fec342d6be2413a873c38e5b896e70b21ebb3fa32197a10ebb49f59167087c: Status 404 returned error can't find the container with id e7fec342d6be2413a873c38e5b896e70b21ebb3fa32197a10ebb49f59167087c
Sep 08 04:09:33.592217 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:33.592147    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vkrsg"]
Sep 08 04:09:33.594061 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:09:33.594037    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7279f48_120e_42aa_80aa_259fc7f2ae5c.slice/crio-02b9b5bfb967a960093d92a27d30ad8f9525f827b0e835a473f1d1fc60cf7d94 WatchSource:0}: Error finding container 02b9b5bfb967a960093d92a27d30ad8f9525f827b0e835a473f1d1fc60cf7d94: Status 404 returned error can't find the container with id 02b9b5bfb967a960093d92a27d30ad8f9525f827b0e835a473f1d1fc60cf7d94
Sep 08 04:09:34.210800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:34.210757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vkrsg" event={"ID":"b7279f48-120e-42aa-80aa-259fc7f2ae5c","Type":"ContainerStarted","Data":"02b9b5bfb967a960093d92a27d30ad8f9525f827b0e835a473f1d1fc60cf7d94"}
Sep 08 04:09:34.212259 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:34.212226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-sgfxt" event={"ID":"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59","Type":"ContainerStarted","Data":"e7fec342d6be2413a873c38e5b896e70b21ebb3fa32197a10ebb49f59167087c"}
Sep 08 04:09:37.235022 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.234980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vkrsg" event={"ID":"b7279f48-120e-42aa-80aa-259fc7f2ae5c","Type":"ContainerStarted","Data":"b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c"}
Sep 08 04:09:37.236635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.236608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-sgfxt" event={"ID":"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59","Type":"ContainerStarted","Data":"9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f"}
Sep 08 04:09:37.237993 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.237973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl" event={"ID":"957ca297-c9b6-4c20-8ed3-149731a44ee4","Type":"ContainerStarted","Data":"d59760484afc52372a2445455e6d99d11f450bbeec6646a521eb7d5a2ef96e69"}
Sep 08 04:09:37.238072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.238039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:09:37.251789 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.251714    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-vkrsg" podStartSLOduration=2.27823782 podStartE2EDuration="5.251703507s" totalImagesPullingTime="2.973465687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:09:37.250934163 +0000 UTC m=+557.609413723" watchObservedRunningTime="2026-09-08 04:09:37.251703507 +0000 UTC m=+557.610183068"
Sep 08 04:09:37.267491 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.267443    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl" podStartSLOduration=1.780268017 podStartE2EDuration="5.267427854s" totalImagesPullingTime="3.487159837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:09:37.265766593 +0000 UTC m=+557.624246154" watchObservedRunningTime="2026-09-08 04:09:37.267427854 +0000 UTC m=+557.625907415"
Sep 08 04:09:37.284139 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.284115    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-sgfxt"]
Sep 08 04:09:37.292075 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:37.292031    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-sgfxt" podStartSLOduration=2.156721492 podStartE2EDuration="5.292016861s" totalImagesPullingTime="3.135295369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:09:37.291732633 +0000 UTC m=+557.650212194" watchObservedRunningTime="2026-09-08 04:09:37.292016861 +0000 UTC m=+557.650496424"
Sep 08 04:09:39.245537 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:39.245497    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-sgfxt" podUID="2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" containerName="authorino" containerID="cri-o://9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f" gracePeriod=30
Sep 08 04:09:39.481680 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:39.481659    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:39.560515 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:39.560493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjd5q\" (UniqueName: \"kubernetes.io/projected/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59-kube-api-access-tjd5q\") pod \"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59\" (UID: \"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59\") "
Sep 08 04:09:39.562431 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:39.562408    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59-kube-api-access-tjd5q" (OuterVolumeSpecName: "kube-api-access-tjd5q") pod "2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" (UID: "2af5f056-99c7-4a69-96fb-fbdb1ac0ac59"). InnerVolumeSpecName "kube-api-access-tjd5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:09:39.661063 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:39.661041    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjd5q\" (UniqueName: \"kubernetes.io/projected/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59-kube-api-access-tjd5q\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:09:40.250254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.250224    2583 generic.go:356] "Generic (PLEG): container finished" podID="2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" containerID="9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f" exitCode=0
Sep 08 04:09:40.250584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.250260    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-sgfxt"
Sep 08 04:09:40.250584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.250307    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-sgfxt" event={"ID":"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59","Type":"ContainerDied","Data":"9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f"}
Sep 08 04:09:40.250584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.250338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-sgfxt" event={"ID":"2af5f056-99c7-4a69-96fb-fbdb1ac0ac59","Type":"ContainerDied","Data":"e7fec342d6be2413a873c38e5b896e70b21ebb3fa32197a10ebb49f59167087c"}
Sep 08 04:09:40.250584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.250358    2583 scope.go:117] "RemoveContainer" containerID="9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f"
Sep 08 04:09:40.258150 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.258134    2583 scope.go:117] "RemoveContainer" containerID="9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f"
Sep 08 04:09:40.258400 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:09:40.258381    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f\": container with ID starting with 9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f not found: ID does not exist" containerID="9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f"
Sep 08 04:09:40.258439 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.258409    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f"} err="failed to get container status \"9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f\": rpc error: code = NotFound desc = could not find container \"9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f\": container with ID starting with 9e1e4b62d2a77085fdc08040d7514d9ad247b508670d0c6c52348042f58e579f not found: ID does not exist"
Sep 08 04:09:40.267080 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.267056    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-sgfxt"]
Sep 08 04:09:40.270851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:40.270834    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-sgfxt"]
Sep 08 04:09:42.158584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:42.158550    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" path="/var/lib/kubelet/pods/2af5f056-99c7-4a69-96fb-fbdb1ac0ac59/volumes"
Sep 08 04:09:48.242127 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:09:48.242090    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"
Sep 08 04:10:02.943686 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:02.943651    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qlvks"]
Sep 08 04:10:02.944117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:02.943943    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" containerName="authorino"
Sep 08 04:10:02.944117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:02.943955    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" containerName="authorino"
Sep 08 04:10:02.944117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:02.944008    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af5f056-99c7-4a69-96fb-fbdb1ac0ac59" containerName="authorino"
Sep 08 04:10:02.946833 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:02.946813    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:02.955829 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:02.955806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qlvks"]
Sep 08 04:10:03.014657 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.014623    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7xk\" (UniqueName: \"kubernetes.io/projected/f0251520-cd8a-4e9b-8011-a185237cb46e-kube-api-access-5z7xk\") pod \"authorino-5dc7ccb5f-qlvks\" (UID: \"f0251520-cd8a-4e9b-8011-a185237cb46e\") " pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:03.114952 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.114927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7xk\" (UniqueName: \"kubernetes.io/projected/f0251520-cd8a-4e9b-8011-a185237cb46e-kube-api-access-5z7xk\") pod \"authorino-5dc7ccb5f-qlvks\" (UID: \"f0251520-cd8a-4e9b-8011-a185237cb46e\") " pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:03.122909 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.122883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7xk\" (UniqueName: \"kubernetes.io/projected/f0251520-cd8a-4e9b-8011-a185237cb46e-kube-api-access-5z7xk\") pod \"authorino-5dc7ccb5f-qlvks\" (UID: \"f0251520-cd8a-4e9b-8011-a185237cb46e\") " pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:03.187072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.187043    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qlvks"]
Sep 08 04:10:03.187204 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.187191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:03.211088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.211063    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-558b96dc7c-924rh"]
Sep 08 04:10:03.214407 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.214388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:03.221156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.220939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558b96dc7c-924rh"]
Sep 08 04:10:03.264875 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.264851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-558b96dc7c-924rh"]
Sep 08 04:10:03.265081 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:10:03.265062    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9tpqv], unattached volumes=[], failed to process volumes=[kube-api-access-9tpqv]: context canceled" pod="kuadrant-system/authorino-558b96dc7c-924rh" podUID="d9107c62-f521-46f0-8022-35c5b62312fb"
Sep 08 04:10:03.291996 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.291975    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85899c496c-bbvlf"]
Sep 08 04:10:03.294959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.294944    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.297118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.297100    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 04:10:03.306199 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.306179    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85899c496c-bbvlf"]
Sep 08 04:10:03.317072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.317050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpqv\" (UniqueName: \"kubernetes.io/projected/d9107c62-f521-46f0-8022-35c5b62312fb-kube-api-access-9tpqv\") pod \"authorino-558b96dc7c-924rh\" (UID: \"d9107c62-f521-46f0-8022-35c5b62312fb\") " pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:03.338401 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.338372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:03.344098 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.344081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:03.418238 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.418214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfbx\" (UniqueName: \"kubernetes.io/projected/c63ba97d-f203-40ce-9bf1-9af5c2473886-kube-api-access-tnfbx\") pod \"authorino-85899c496c-bbvlf\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") " pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.418336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.418286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpqv\" (UniqueName: \"kubernetes.io/projected/d9107c62-f521-46f0-8022-35c5b62312fb-kube-api-access-9tpqv\") pod \"authorino-558b96dc7c-924rh\" (UID: \"d9107c62-f521-46f0-8022-35c5b62312fb\") " pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:03.418392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.418333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c63ba97d-f203-40ce-9bf1-9af5c2473886-tls-cert\") pod \"authorino-85899c496c-bbvlf\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") " pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.425540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.425515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpqv\" (UniqueName: \"kubernetes.io/projected/d9107c62-f521-46f0-8022-35c5b62312fb-kube-api-access-9tpqv\") pod \"authorino-558b96dc7c-924rh\" (UID: \"d9107c62-f521-46f0-8022-35c5b62312fb\") " pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:03.454318 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.454295    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qlvks"]
Sep 08 04:10:03.456788 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:10:03.456767    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0251520_cd8a_4e9b_8011_a185237cb46e.slice/crio-1f2dfb9171e542edea378ccb6b569a482367a64c7c33f1e01222ff90beda990a WatchSource:0}: Error finding container 1f2dfb9171e542edea378ccb6b569a482367a64c7c33f1e01222ff90beda990a: Status 404 returned error can't find the container with id 1f2dfb9171e542edea378ccb6b569a482367a64c7c33f1e01222ff90beda990a
Sep 08 04:10:03.518962 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.518943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tpqv\" (UniqueName: \"kubernetes.io/projected/d9107c62-f521-46f0-8022-35c5b62312fb-kube-api-access-9tpqv\") pod \"d9107c62-f521-46f0-8022-35c5b62312fb\" (UID: \"d9107c62-f521-46f0-8022-35c5b62312fb\") "
Sep 08 04:10:03.519072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.519057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c63ba97d-f203-40ce-9bf1-9af5c2473886-tls-cert\") pod \"authorino-85899c496c-bbvlf\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") " pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.519118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.519089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfbx\" (UniqueName: \"kubernetes.io/projected/c63ba97d-f203-40ce-9bf1-9af5c2473886-kube-api-access-tnfbx\") pod \"authorino-85899c496c-bbvlf\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") " pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.520888 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.520869    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9107c62-f521-46f0-8022-35c5b62312fb-kube-api-access-9tpqv" (OuterVolumeSpecName: "kube-api-access-9tpqv") pod "d9107c62-f521-46f0-8022-35c5b62312fb" (UID: "d9107c62-f521-46f0-8022-35c5b62312fb"). InnerVolumeSpecName "kube-api-access-9tpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:10:03.521266 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.521252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c63ba97d-f203-40ce-9bf1-9af5c2473886-tls-cert\") pod \"authorino-85899c496c-bbvlf\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") " pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.526750 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.526725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfbx\" (UniqueName: \"kubernetes.io/projected/c63ba97d-f203-40ce-9bf1-9af5c2473886-kube-api-access-tnfbx\") pod \"authorino-85899c496c-bbvlf\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") " pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.603790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.603766    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:10:03.619903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.619879    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpqv\" (UniqueName: \"kubernetes.io/projected/d9107c62-f521-46f0-8022-35c5b62312fb-kube-api-access-9tpqv\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:10:03.871493 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:03.871471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85899c496c-bbvlf"]
Sep 08 04:10:03.873706 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:10:03.873682    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc63ba97d_f203_40ce_9bf1_9af5c2473886.slice/crio-ee9216ffc3f5a266f06c320de51d4624bc120221a50a1c8d7993b9c7ee71d80c WatchSource:0}: Error finding container ee9216ffc3f5a266f06c320de51d4624bc120221a50a1c8d7993b9c7ee71d80c: Status 404 returned error can't find the container with id ee9216ffc3f5a266f06c320de51d4624bc120221a50a1c8d7993b9c7ee71d80c
Sep 08 04:10:04.344978 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.344942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85899c496c-bbvlf" event={"ID":"c63ba97d-f203-40ce-9bf1-9af5c2473886","Type":"ContainerStarted","Data":"ee9216ffc3f5a266f06c320de51d4624bc120221a50a1c8d7993b9c7ee71d80c"}
Sep 08 04:10:04.347023 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.346995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks" event={"ID":"f0251520-cd8a-4e9b-8011-a185237cb46e","Type":"ContainerStarted","Data":"d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50"}
Sep 08 04:10:04.347023 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.347004    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks" podUID="f0251520-cd8a-4e9b-8011-a185237cb46e" containerName="authorino" containerID="cri-o://d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50" gracePeriod=30
Sep 08 04:10:04.347205 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.347038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks" event={"ID":"f0251520-cd8a-4e9b-8011-a185237cb46e","Type":"ContainerStarted","Data":"1f2dfb9171e542edea378ccb6b569a482367a64c7c33f1e01222ff90beda990a"}
Sep 08 04:10:04.347205 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.347003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558b96dc7c-924rh"
Sep 08 04:10:04.362047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.361995    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks" podStartSLOduration=1.5502473220000001 podStartE2EDuration="2.361977096s" totalImagesPullingTime="811.729774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:10:04.361468086 +0000 UTC m=+584.719947646" watchObservedRunningTime="2026-09-08 04:10:04.361977096 +0000 UTC m=+584.720456656"
Sep 08 04:10:04.383067 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.383017    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-558b96dc7c-924rh"]
Sep 08 04:10:04.387077 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.387058    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-558b96dc7c-924rh"]
Sep 08 04:10:04.571544 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.571519    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:04.626811 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.626780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z7xk\" (UniqueName: \"kubernetes.io/projected/f0251520-cd8a-4e9b-8011-a185237cb46e-kube-api-access-5z7xk\") pod \"f0251520-cd8a-4e9b-8011-a185237cb46e\" (UID: \"f0251520-cd8a-4e9b-8011-a185237cb46e\") "
Sep 08 04:10:04.628702 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.628680    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0251520-cd8a-4e9b-8011-a185237cb46e-kube-api-access-5z7xk" (OuterVolumeSpecName: "kube-api-access-5z7xk") pod "f0251520-cd8a-4e9b-8011-a185237cb46e" (UID: "f0251520-cd8a-4e9b-8011-a185237cb46e"). InnerVolumeSpecName "kube-api-access-5z7xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:10:04.727349 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:04.727294    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7xk\" (UniqueName: \"kubernetes.io/projected/f0251520-cd8a-4e9b-8011-a185237cb46e-kube-api-access-5z7xk\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:10:05.352182 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.352150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85899c496c-bbvlf" event={"ID":"c63ba97d-f203-40ce-9bf1-9af5c2473886","Type":"ContainerStarted","Data":"cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1"}
Sep 08 04:10:05.353431 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.353406    2583 generic.go:356] "Generic (PLEG): container finished" podID="f0251520-cd8a-4e9b-8011-a185237cb46e" containerID="d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50" exitCode=0
Sep 08 04:10:05.353548 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.353434    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks"
Sep 08 04:10:05.353548 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.353451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks" event={"ID":"f0251520-cd8a-4e9b-8011-a185237cb46e","Type":"ContainerDied","Data":"d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50"}
Sep 08 04:10:05.353548 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.353472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-qlvks" event={"ID":"f0251520-cd8a-4e9b-8011-a185237cb46e","Type":"ContainerDied","Data":"1f2dfb9171e542edea378ccb6b569a482367a64c7c33f1e01222ff90beda990a"}
Sep 08 04:10:05.353548 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.353487    2583 scope.go:117] "RemoveContainer" containerID="d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50"
Sep 08 04:10:05.366157 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.366136    2583 scope.go:117] "RemoveContainer" containerID="d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50"
Sep 08 04:10:05.366429 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:10:05.366410    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50\": container with ID starting with d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50 not found: ID does not exist" containerID="d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50"
Sep 08 04:10:05.366482 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.366436    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50"} err="failed to get container status \"d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50\": rpc error: code = NotFound desc = could not find container \"d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50\": container with ID starting with d2db005309866410f690eb298e0a43003c4b894b53cbdb4f46c27c7c2a062f50 not found: ID does not exist"
Sep 08 04:10:05.374905 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.374366    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85899c496c-bbvlf" podStartSLOduration=1.7485236 podStartE2EDuration="2.374350094s" totalImagesPullingTime="625.826494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:10:05.372387892 +0000 UTC m=+585.730867454" watchObservedRunningTime="2026-09-08 04:10:05.374350094 +0000 UTC m=+585.732829657"
Sep 08 04:10:05.388353 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.388302    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qlvks"]
Sep 08 04:10:05.393883 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.393862    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-qlvks"]
Sep 08 04:10:05.398488 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.398466    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vkrsg"]
Sep 08 04:10:05.398625 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.398605    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-vkrsg" podUID="b7279f48-120e-42aa-80aa-259fc7f2ae5c" containerName="authorino" containerID="cri-o://b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c" gracePeriod=30
Sep 08 04:10:05.624112 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.624088    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:10:05.734697 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.734638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmcb7\" (UniqueName: \"kubernetes.io/projected/b7279f48-120e-42aa-80aa-259fc7f2ae5c-kube-api-access-nmcb7\") pod \"b7279f48-120e-42aa-80aa-259fc7f2ae5c\" (UID: \"b7279f48-120e-42aa-80aa-259fc7f2ae5c\") "
Sep 08 04:10:05.736590 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.736567    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7279f48-120e-42aa-80aa-259fc7f2ae5c-kube-api-access-nmcb7" (OuterVolumeSpecName: "kube-api-access-nmcb7") pod "b7279f48-120e-42aa-80aa-259fc7f2ae5c" (UID: "b7279f48-120e-42aa-80aa-259fc7f2ae5c"). InnerVolumeSpecName "kube-api-access-nmcb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:10:05.836054 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:05.836028    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcb7\" (UniqueName: \"kubernetes.io/projected/b7279f48-120e-42aa-80aa-259fc7f2ae5c-kube-api-access-nmcb7\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:10:06.158838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.158803    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9107c62-f521-46f0-8022-35c5b62312fb" path="/var/lib/kubelet/pods/d9107c62-f521-46f0-8022-35c5b62312fb/volumes"
Sep 08 04:10:06.159047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.159035    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0251520-cd8a-4e9b-8011-a185237cb46e" path="/var/lib/kubelet/pods/f0251520-cd8a-4e9b-8011-a185237cb46e/volumes"
Sep 08 04:10:06.358192 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.358167    2583 generic.go:356] "Generic (PLEG): container finished" podID="b7279f48-120e-42aa-80aa-259fc7f2ae5c" containerID="b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c" exitCode=0
Sep 08 04:10:06.358536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.358209    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vkrsg"
Sep 08 04:10:06.358536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.358242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vkrsg" event={"ID":"b7279f48-120e-42aa-80aa-259fc7f2ae5c","Type":"ContainerDied","Data":"b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c"}
Sep 08 04:10:06.358536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.358277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vkrsg" event={"ID":"b7279f48-120e-42aa-80aa-259fc7f2ae5c","Type":"ContainerDied","Data":"02b9b5bfb967a960093d92a27d30ad8f9525f827b0e835a473f1d1fc60cf7d94"}
Sep 08 04:10:06.358536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.358299    2583 scope.go:117] "RemoveContainer" containerID="b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c"
Sep 08 04:10:06.365927 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.365907    2583 scope.go:117] "RemoveContainer" containerID="b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c"
Sep 08 04:10:06.366165 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:10:06.366148    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c\": container with ID starting with b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c not found: ID does not exist" containerID="b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c"
Sep 08 04:10:06.366232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.366170    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c"} err="failed to get container status \"b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c\": rpc error: code = NotFound desc = could not find container \"b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c\": container with ID starting with b6671a4fca1785377f5da5fa052050f9e8471afdb19def33cb78f1747b314e6c not found: ID does not exist"
Sep 08 04:10:06.375102 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.375080    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vkrsg"]
Sep 08 04:10:06.378695 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:06.378673    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vkrsg"]
Sep 08 04:10:08.159021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:08.158994    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7279f48-120e-42aa-80aa-259fc7f2ae5c" path="/var/lib/kubelet/pods/b7279f48-120e-42aa-80aa-259fc7f2ae5c/volumes"
Sep 08 04:10:20.094772 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:20.094726    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:10:20.096427 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:20.096404    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:10:47.868899 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:10:47.868819    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:11:12.061266 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:12.061225    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:11:31.443339 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:31.443286    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:11:44.395747 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:44.395713    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:11:49.942714 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:49.942672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:11:57.555920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:57.555885    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:11:58.826159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826129    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"]
Sep 08 04:11:58.826511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826395    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7279f48-120e-42aa-80aa-259fc7f2ae5c" containerName="authorino"
Sep 08 04:11:58.826511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826405    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7279f48-120e-42aa-80aa-259fc7f2ae5c" containerName="authorino"
Sep 08 04:11:58.826511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826417    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0251520-cd8a-4e9b-8011-a185237cb46e" containerName="authorino"
Sep 08 04:11:58.826511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826422    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0251520-cd8a-4e9b-8011-a185237cb46e" containerName="authorino"
Sep 08 04:11:58.826511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826471    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7279f48-120e-42aa-80aa-259fc7f2ae5c" containerName="authorino"
Sep 08 04:11:58.826511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.826477    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0251520-cd8a-4e9b-8011-a185237cb46e" containerName="authorino"
Sep 08 04:11:58.829474 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.829456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:58.832692 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.832670    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ts7g5\""
Sep 08 04:11:58.832692 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.832682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 04:11:58.832862 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.832703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 04:11:58.832862 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.832690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 04:11:58.839557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.839537    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"]
Sep 08 04:11:58.918193 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.918172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:58.918274 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.918203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:58.918274 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.918229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:58.918353 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.918288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:58.918353 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.918328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:58.918353 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:58.918350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w74qz\" (UniqueName: \"kubernetes.io/projected/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-kube-api-access-w74qz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019145 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w74qz\" (UniqueName: \"kubernetes.io/projected/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-kube-api-access-w74qz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019145 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019233 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019233 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019233 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019412 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019473 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.019593 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.019574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.021329 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.021304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.021683 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.021666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.029325 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.029300    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w74qz\" (UniqueName: \"kubernetes.io/projected/0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05-kube-api-access-w74qz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jgl67\" (UID: \"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.139468 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.139409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:11:59.409369 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.409345    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"]
Sep 08 04:11:59.411604 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:11:59.411575    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab44fd5_a0fb_4fe5_b9f9_6311204f8a05.slice/crio-cdf81323c25691255829abcc6c0b5923e8af1d735a1c42d8e28b1c3349245a7b WatchSource:0}: Error finding container cdf81323c25691255829abcc6c0b5923e8af1d735a1c42d8e28b1c3349245a7b: Status 404 returned error can't find the container with id cdf81323c25691255829abcc6c0b5923e8af1d735a1c42d8e28b1c3349245a7b
Sep 08 04:11:59.417907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.417882    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 04:11:59.811113 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:11:59.811071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67" event={"ID":"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05","Type":"ContainerStarted","Data":"cdf81323c25691255829abcc6c0b5923e8af1d735a1c42d8e28b1c3349245a7b"}
Sep 08 04:12:00.817028 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:00.816989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67" event={"ID":"0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05","Type":"ContainerStarted","Data":"579a65ff190d923991adf369c4b5220b4060c35da43a497e60d9ce4c51c25ec5"}
Sep 08 04:12:00.834634 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:00.834588    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67" podStartSLOduration=1.815860259 podStartE2EDuration="2.83457215s" totalImagesPullingTime="1.018711891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:12:00.833485626 +0000 UTC m=+701.191965186" watchObservedRunningTime="2026-09-08 04:12:00.83457215 +0000 UTC m=+701.193051716"
Sep 08 04:12:04.748104 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:04.748071    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:12:09.139785 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:09.139755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:12:09.140228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:09.139815    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:12:09.152289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:09.152263    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:12:09.863231 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:09.863206    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jgl67"
Sep 08 04:12:11.627338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.627299    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"]
Sep 08 04:12:11.630455 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.630431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.632632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.632609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 04:12:11.639083 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.639060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"]
Sep 08 04:12:11.703799 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.703772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.703947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.703802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.703947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.703830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.703947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.703847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774282f2-e389-46a1-9001-71537cb4c482-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.703947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.703867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.703947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.703926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bj5\" (UniqueName: \"kubernetes.io/projected/774282f2-e389-46a1-9001-71537cb4c482-kube-api-access-j4bj5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805130 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805130 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774282f2-e389-46a1-9001-71537cb4c482-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bj5\" (UniqueName: \"kubernetes.io/projected/774282f2-e389-46a1-9001-71537cb4c482-kube-api-access-j4bj5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805465 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805597 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805597 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.805712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.805637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.807467 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.807447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774282f2-e389-46a1-9001-71537cb4c482-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.807773 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.807734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774282f2-e389-46a1-9001-71537cb4c482-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.813012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.812989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bj5\" (UniqueName: \"kubernetes.io/projected/774282f2-e389-46a1-9001-71537cb4c482-kube-api-access-j4bj5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh\" (UID: \"774282f2-e389-46a1-9001-71537cb4c482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:11.940222 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:11.940171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:12.206755 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:12.206694    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"]
Sep 08 04:12:12.208946 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:12:12.208921    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod774282f2_e389_46a1_9001_71537cb4c482.slice/crio-7b0c721e25164e7984d77772b2a7652c38ee9c7c2cd5ec4232ea4ec9faf7dddc WatchSource:0}: Error finding container 7b0c721e25164e7984d77772b2a7652c38ee9c7c2cd5ec4232ea4ec9faf7dddc: Status 404 returned error can't find the container with id 7b0c721e25164e7984d77772b2a7652c38ee9c7c2cd5ec4232ea4ec9faf7dddc
Sep 08 04:12:12.867386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:12.867323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh" event={"ID":"774282f2-e389-46a1-9001-71537cb4c482","Type":"ContainerStarted","Data":"1e49cb09dad495746fa7b8ee6711212a2c16c140b18ca9cfa882c41c6a90c926"}
Sep 08 04:12:12.867386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:12.867363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh" event={"ID":"774282f2-e389-46a1-9001-71537cb4c482","Type":"ContainerStarted","Data":"7b0c721e25164e7984d77772b2a7652c38ee9c7c2cd5ec4232ea4ec9faf7dddc"}
Sep 08 04:12:12.884166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:12.884118    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh" podStartSLOduration=1.704213212 podStartE2EDuration="1.884104523s" totalImagesPullingTime="179.891311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:12:12.883204092 +0000 UTC m=+713.241683652" watchObservedRunningTime="2026-09-08 04:12:12.884104523 +0000 UTC m=+713.242584082"
Sep 08 04:12:13.960605 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:13.960565    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:12:21.940260 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:21.940225    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:21.940680 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:21.940277    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:21.953251 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:21.953223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:12:22.914276 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:12:22.914242    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh"
Sep 08 04:13:09.308882 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.308848    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-574c4d4cdb-ttn42"]
Sep 08 04:13:09.311955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.311937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.323015 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.322982    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-574c4d4cdb-ttn42"]
Sep 08 04:13:09.492450 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.492421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/703da780-afd1-4e25-a0ed-e9946fb4d4b8-tls-cert\") pod \"authorino-574c4d4cdb-ttn42\" (UID: \"703da780-afd1-4e25-a0ed-e9946fb4d4b8\") " pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.492570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.492467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psnww\" (UniqueName: \"kubernetes.io/projected/703da780-afd1-4e25-a0ed-e9946fb4d4b8-kube-api-access-psnww\") pod \"authorino-574c4d4cdb-ttn42\" (UID: \"703da780-afd1-4e25-a0ed-e9946fb4d4b8\") " pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.593225 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.593166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psnww\" (UniqueName: \"kubernetes.io/projected/703da780-afd1-4e25-a0ed-e9946fb4d4b8-kube-api-access-psnww\") pod \"authorino-574c4d4cdb-ttn42\" (UID: \"703da780-afd1-4e25-a0ed-e9946fb4d4b8\") " pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.593225 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.593219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/703da780-afd1-4e25-a0ed-e9946fb4d4b8-tls-cert\") pod \"authorino-574c4d4cdb-ttn42\" (UID: \"703da780-afd1-4e25-a0ed-e9946fb4d4b8\") " pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.595427 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.595402    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/703da780-afd1-4e25-a0ed-e9946fb4d4b8-tls-cert\") pod \"authorino-574c4d4cdb-ttn42\" (UID: \"703da780-afd1-4e25-a0ed-e9946fb4d4b8\") " pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.601751 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.601720    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psnww\" (UniqueName: \"kubernetes.io/projected/703da780-afd1-4e25-a0ed-e9946fb4d4b8-kube-api-access-psnww\") pod \"authorino-574c4d4cdb-ttn42\" (UID: \"703da780-afd1-4e25-a0ed-e9946fb4d4b8\") " pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.622262 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.622244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-574c4d4cdb-ttn42"
Sep 08 04:13:09.888507 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:09.888478    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-574c4d4cdb-ttn42"]
Sep 08 04:13:09.890772 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:13:09.890730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod703da780_afd1_4e25_a0ed_e9946fb4d4b8.slice/crio-7fe0688a7423814645019a49bdad920833b50e4fac69152add0283037a25a90d WatchSource:0}: Error finding container 7fe0688a7423814645019a49bdad920833b50e4fac69152add0283037a25a90d: Status 404 returned error can't find the container with id 7fe0688a7423814645019a49bdad920833b50e4fac69152add0283037a25a90d
Sep 08 04:13:10.094090 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:10.094051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-574c4d4cdb-ttn42" event={"ID":"703da780-afd1-4e25-a0ed-e9946fb4d4b8","Type":"ContainerStarted","Data":"7fe0688a7423814645019a49bdad920833b50e4fac69152add0283037a25a90d"}
Sep 08 04:13:12.103954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.103925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-574c4d4cdb-ttn42" event={"ID":"703da780-afd1-4e25-a0ed-e9946fb4d4b8","Type":"ContainerStarted","Data":"01a3f93986b9ee1963abfa9fe99d30469caa0220319f46d77fe43d1d7cf20412"}
Sep 08 04:13:12.120555 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.120502    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-574c4d4cdb-ttn42" podStartSLOduration=1.004164713 podStartE2EDuration="3.120483981s" totalImagesPullingTime="2.116319268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:12.11901385 +0000 UTC m=+772.477493411" watchObservedRunningTime="2026-09-08 04:13:12.120483981 +0000 UTC m=+772.478963535"
Sep 08 04:13:12.149028 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.148991    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85899c496c-bbvlf"]
Sep 08 04:13:12.150723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.150682    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85899c496c-bbvlf" podUID="c63ba97d-f203-40ce-9bf1-9af5c2473886" containerName="authorino" containerID="cri-o://cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1" gracePeriod=30
Sep 08 04:13:12.403416 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.403376    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:13:12.519007 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.518923    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnfbx\" (UniqueName: \"kubernetes.io/projected/c63ba97d-f203-40ce-9bf1-9af5c2473886-kube-api-access-tnfbx\") pod \"c63ba97d-f203-40ce-9bf1-9af5c2473886\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") "
Sep 08 04:13:12.519007 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.518978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c63ba97d-f203-40ce-9bf1-9af5c2473886-tls-cert\") pod \"c63ba97d-f203-40ce-9bf1-9af5c2473886\" (UID: \"c63ba97d-f203-40ce-9bf1-9af5c2473886\") "
Sep 08 04:13:12.521099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.521062    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63ba97d-f203-40ce-9bf1-9af5c2473886-kube-api-access-tnfbx" (OuterVolumeSpecName: "kube-api-access-tnfbx") pod "c63ba97d-f203-40ce-9bf1-9af5c2473886" (UID: "c63ba97d-f203-40ce-9bf1-9af5c2473886"). InnerVolumeSpecName "kube-api-access-tnfbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:13:12.528999 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.528967    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63ba97d-f203-40ce-9bf1-9af5c2473886-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c63ba97d-f203-40ce-9bf1-9af5c2473886" (UID: "c63ba97d-f203-40ce-9bf1-9af5c2473886"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:13:12.619893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.619846    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnfbx\" (UniqueName: \"kubernetes.io/projected/c63ba97d-f203-40ce-9bf1-9af5c2473886-kube-api-access-tnfbx\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:13:12.619893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:12.619879    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c63ba97d-f203-40ce-9bf1-9af5c2473886-tls-cert\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:13:13.111353 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.111321    2583 generic.go:356] "Generic (PLEG): container finished" podID="c63ba97d-f203-40ce-9bf1-9af5c2473886" containerID="cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1" exitCode=0
Sep 08 04:13:13.111800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.111374    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85899c496c-bbvlf"
Sep 08 04:13:13.111800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.111415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85899c496c-bbvlf" event={"ID":"c63ba97d-f203-40ce-9bf1-9af5c2473886","Type":"ContainerDied","Data":"cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1"}
Sep 08 04:13:13.111800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.111458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85899c496c-bbvlf" event={"ID":"c63ba97d-f203-40ce-9bf1-9af5c2473886","Type":"ContainerDied","Data":"ee9216ffc3f5a266f06c320de51d4624bc120221a50a1c8d7993b9c7ee71d80c"}
Sep 08 04:13:13.111800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.111476    2583 scope.go:117] "RemoveContainer" containerID="cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1"
Sep 08 04:13:13.119803 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.119785    2583 scope.go:117] "RemoveContainer" containerID="cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1"
Sep 08 04:13:13.120128 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:13.120106    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1\": container with ID starting with cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1 not found: ID does not exist" containerID="cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1"
Sep 08 04:13:13.120225 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.120134    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1"} err="failed to get container status \"cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1\": rpc error: code = NotFound desc = could not find container \"cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1\": container with ID starting with cb0f8910499258d211a1d86cebceaf478bdfdce9e1b6eba0176a5118a8f343e1 not found: ID does not exist"
Sep 08 04:13:13.138952 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.138922    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85899c496c-bbvlf"]
Sep 08 04:13:13.144091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:13.144069    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85899c496c-bbvlf"]
Sep 08 04:13:14.160619 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:14.160586    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c63ba97d-f203-40ce-9bf1-9af5c2473886" path="/var/lib/kubelet/pods/c63ba97d-f203-40ce-9bf1-9af5c2473886/volumes"
Sep 08 04:13:30.678448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.678415    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:30.678821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.678713    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63ba97d-f203-40ce-9bf1-9af5c2473886" containerName="authorino"
Sep 08 04:13:30.678821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.678724    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63ba97d-f203-40ce-9bf1-9af5c2473886" containerName="authorino"
Sep 08 04:13:30.678821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.678794    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c63ba97d-f203-40ce-9bf1-9af5c2473886" containerName="authorino"
Sep 08 04:13:30.682477 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.682460    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:30.684760 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.684723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nwvtr\""
Sep 08 04:13:30.688255 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.688231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:30.758635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.758615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4d9q\" (UniqueName: \"kubernetes.io/projected/b32f5744-b24a-4d20-a095-381855668d9c-kube-api-access-h4d9q\") pod \"test-curl-7852-140344018705200\" (UID: \"b32f5744-b24a-4d20-a095-381855668d9c\") " pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:30.859833 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.859802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4d9q\" (UniqueName: \"kubernetes.io/projected/b32f5744-b24a-4d20-a095-381855668d9c-kube-api-access-h4d9q\") pod \"test-curl-7852-140344018705200\" (UID: \"b32f5744-b24a-4d20-a095-381855668d9c\") " pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:30.867586 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.867566    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4d9q\" (UniqueName: \"kubernetes.io/projected/b32f5744-b24a-4d20-a095-381855668d9c-kube-api-access-h4d9q\") pod \"test-curl-7852-140344018705200\" (UID: \"b32f5744-b24a-4d20-a095-381855668d9c\") " pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:30.992048 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:30.991999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:31.260126 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:31.259944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:31.262086 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:13:31.262051    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb32f5744_b24a_4d20_a095_381855668d9c.slice/crio-ff91bbf762ff3016e54f723fc6224c27402a2e2dd6a906e0cabe310dbdb7c8a4 WatchSource:0}: Error finding container ff91bbf762ff3016e54f723fc6224c27402a2e2dd6a906e0cabe310dbdb7c8a4: Status 404 returned error can't find the container with id ff91bbf762ff3016e54f723fc6224c27402a2e2dd6a906e0cabe310dbdb7c8a4
Sep 08 04:13:32.196068 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:32.196031    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140344018705200" event={"ID":"b32f5744-b24a-4d20-a095-381855668d9c","Type":"ContainerStarted","Data":"ff91bbf762ff3016e54f723fc6224c27402a2e2dd6a906e0cabe310dbdb7c8a4"}
Sep 08 04:13:32.854733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:32.854700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:13:34.206272 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:34.206234    2583 generic.go:356] "Generic (PLEG): container finished" podID="b32f5744-b24a-4d20-a095-381855668d9c" containerID="15cde86d3f6be804e5fc4e05e5a1282ad5a791f14dbf2c57eaedc678fbcc306e" exitCode=0
Sep 08 04:13:34.206272 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:34.206275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140344018705200" event={"ID":"b32f5744-b24a-4d20-a095-381855668d9c","Type":"ContainerDied","Data":"15cde86d3f6be804e5fc4e05e5a1282ad5a791f14dbf2c57eaedc678fbcc306e"}
Sep 08 04:13:35.331730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.331704    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:35.365808 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.365765    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140344018705200_b32f5744-b24a-4d20-a095-381855668d9c/test-curl-7852-140344018705200/0.log"
Sep 08 04:13:35.384251 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.384226    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:35.385967 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.385949    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:35.400042 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.400016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4d9q\" (UniqueName: \"kubernetes.io/projected/b32f5744-b24a-4d20-a095-381855668d9c-kube-api-access-h4d9q\") pod \"b32f5744-b24a-4d20-a095-381855668d9c\" (UID: \"b32f5744-b24a-4d20-a095-381855668d9c\") "
Sep 08 04:13:35.402016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.401998    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32f5744-b24a-4d20-a095-381855668d9c-kube-api-access-h4d9q" (OuterVolumeSpecName: "kube-api-access-h4d9q") pod "b32f5744-b24a-4d20-a095-381855668d9c" (UID: "b32f5744-b24a-4d20-a095-381855668d9c"). InnerVolumeSpecName "kube-api-access-h4d9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:13:35.500608 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.500588    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4d9q\" (UniqueName: \"kubernetes.io/projected/b32f5744-b24a-4d20-a095-381855668d9c-kube-api-access-h4d9q\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:13:35.501606 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.501589    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:35.501878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.501866    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32f5744-b24a-4d20-a095-381855668d9c" containerName="test-curl-7852-140344018705200"
Sep 08 04:13:35.501913 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.501880    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32f5744-b24a-4d20-a095-381855668d9c" containerName="test-curl-7852-140344018705200"
Sep 08 04:13:35.501950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.501936    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32f5744-b24a-4d20-a095-381855668d9c" containerName="test-curl-7852-140344018705200"
Sep 08 04:13:35.504641 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.504627    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:35.510072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.510053    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:35.601299 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.601275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47t82\" (UniqueName: \"kubernetes.io/projected/6c634844-228b-48ee-8d0f-969ebcda05de-kube-api-access-47t82\") pod \"test-curl-7852-140344018705200\" (UID: \"6c634844-228b-48ee-8d0f-969ebcda05de\") " pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:35.701920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.701894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47t82\" (UniqueName: \"kubernetes.io/projected/6c634844-228b-48ee-8d0f-969ebcda05de-kube-api-access-47t82\") pod \"test-curl-7852-140344018705200\" (UID: \"6c634844-228b-48ee-8d0f-969ebcda05de\") " pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:35.709953 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.709929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47t82\" (UniqueName: \"kubernetes.io/projected/6c634844-228b-48ee-8d0f-969ebcda05de-kube-api-access-47t82\") pod \"test-curl-7852-140344018705200\" (UID: \"6c634844-228b-48ee-8d0f-969ebcda05de\") " pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:35.814109 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:35.814063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:36.078831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:36.078761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:36.081457 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:13:36.081429    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c634844_228b_48ee_8d0f_969ebcda05de.slice/crio-9646cca69bfee425adba9fdad4632014d7b12cf6edc6c4ab2d0ed2ac80a224e9 WatchSource:0}: Error finding container 9646cca69bfee425adba9fdad4632014d7b12cf6edc6c4ab2d0ed2ac80a224e9: Status 404 returned error can't find the container with id 9646cca69bfee425adba9fdad4632014d7b12cf6edc6c4ab2d0ed2ac80a224e9
Sep 08 04:13:36.159131 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:36.159101    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32f5744-b24a-4d20-a095-381855668d9c" path="/var/lib/kubelet/pods/b32f5744-b24a-4d20-a095-381855668d9c/volumes"
Sep 08 04:13:36.214567 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:36.214539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140344018705200" event={"ID":"6c634844-228b-48ee-8d0f-969ebcda05de","Type":"ContainerStarted","Data":"9646cca69bfee425adba9fdad4632014d7b12cf6edc6c4ab2d0ed2ac80a224e9"}
Sep 08 04:13:36.215629 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:36.215610    2583 scope.go:117] "RemoveContainer" containerID="15cde86d3f6be804e5fc4e05e5a1282ad5a791f14dbf2c57eaedc678fbcc306e"
Sep 08 04:13:36.215629 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:36.215623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:37.221225 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:37.221162    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c634844-228b-48ee-8d0f-969ebcda05de" containerID="2dd017a064476c9738796d298329758b86e99f142b13dc06c1b430a71f305145" exitCode=0
Sep 08 04:13:37.221540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:37.221249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140344018705200" event={"ID":"6c634844-228b-48ee-8d0f-969ebcda05de","Type":"ContainerDied","Data":"2dd017a064476c9738796d298329758b86e99f142b13dc06c1b430a71f305145"}
Sep 08 04:13:38.355018 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.354993    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:38.377667 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.377629    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140344018705200_6c634844-228b-48ee-8d0f-969ebcda05de/test-curl-7852-140344018705200/0.log"
Sep 08 04:13:38.396930 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.396906    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:38.400645 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.400622    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140344018705200"]
Sep 08 04:13:38.422791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.422763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47t82\" (UniqueName: \"kubernetes.io/projected/6c634844-228b-48ee-8d0f-969ebcda05de-kube-api-access-47t82\") pod \"6c634844-228b-48ee-8d0f-969ebcda05de\" (UID: \"6c634844-228b-48ee-8d0f-969ebcda05de\") "
Sep 08 04:13:38.424865 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.424842    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c634844-228b-48ee-8d0f-969ebcda05de-kube-api-access-47t82" (OuterVolumeSpecName: "kube-api-access-47t82") pod "6c634844-228b-48ee-8d0f-969ebcda05de" (UID: "6c634844-228b-48ee-8d0f-969ebcda05de"). InnerVolumeSpecName "kube-api-access-47t82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:13:38.524041 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:38.524016    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47t82\" (UniqueName: \"kubernetes.io/projected/6c634844-228b-48ee-8d0f-969ebcda05de-kube-api-access-47t82\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:13:39.232239 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:39.232206    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:39.232398 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:39.232219    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9646cca69bfee425adba9fdad4632014d7b12cf6edc6c4ab2d0ed2ac80a224e9"
Sep 08 04:13:39.241724 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:39.241696    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7852-140344018705200\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="6c634844-228b-48ee-8d0f-969ebcda05de" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:40.158735 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:40.158703    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7852-140344018705200\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="6c634844-228b-48ee-8d0f-969ebcda05de" pod="opendatahub/test-curl-7852-140344018705200"
Sep 08 04:13:40.159498 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:40.159476    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c634844-228b-48ee-8d0f-969ebcda05de" path="/var/lib/kubelet/pods/6c634844-228b-48ee-8d0f-969ebcda05de/volumes"
Sep 08 04:13:41.996321 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:41.996289    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"]
Sep 08 04:13:41.996668 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:41.996588    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c634844-228b-48ee-8d0f-969ebcda05de" containerName="test-curl-7852-140344018705200"
Sep 08 04:13:41.996668 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:41.996602    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c634844-228b-48ee-8d0f-969ebcda05de" containerName="test-curl-7852-140344018705200"
Sep 08 04:13:41.996668 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:41.996666    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c634844-228b-48ee-8d0f-969ebcda05de" containerName="test-curl-7852-140344018705200"
Sep 08 04:13:41.999783 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:41.999764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.003189 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.003168    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8d710ebd\""
Sep 08 04:13:42.003286 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.003251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8d710ebd-dockercfg-j2dt4\""
Sep 08 04:13:42.014071 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.014052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"]
Sep 08 04:13:42.040534 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.040488    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"]
Sep 08 04:13:42.044720 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.044700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.050603 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.050581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.050672 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.050615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2252\" (UniqueName: \"kubernetes.io/projected/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-kube-api-access-w2252\") pod \"payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.054397 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.054377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"]
Sep 08 04:13:42.151033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.151011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3b016-ca00-455d-b39d-5647a6b113d1-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") " pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.151151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.151049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.151151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.151069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2252\" (UniqueName: \"kubernetes.io/projected/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-kube-api-access-w2252\") pod \"payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.151151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.151141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7sr\" (UniqueName: \"kubernetes.io/projected/fae3b016-ca00-455d-b39d-5647a6b113d1-kube-api-access-gf7sr\") pod \"payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") " pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.151723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.151699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.160257 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.160230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2252\" (UniqueName: \"kubernetes.io/projected/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-kube-api-access-w2252\") pod \"payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.251794 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.251766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7sr\" (UniqueName: \"kubernetes.io/projected/fae3b016-ca00-455d-b39d-5647a6b113d1-kube-api-access-gf7sr\") pod \"payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") " pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.251873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.251819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3b016-ca00-455d-b39d-5647a6b113d1-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") " pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.252485 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.252462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3b016-ca00-455d-b39d-5647a6b113d1-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") " pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.259013 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.258993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7sr\" (UniqueName: \"kubernetes.io/projected/fae3b016-ca00-455d-b39d-5647a6b113d1-kube-api-access-gf7sr\") pod \"payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") " pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.308770 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.308727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:42.354958 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.354919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:42.578247 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.578193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"]
Sep 08 04:13:42.580441 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:13:42.580419    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9a40f3_8ca0_4758_9fbb_e6f6246719c6.slice/crio-957dc4c2f263cfe8707cbe5d5f1d01c42da86893e6b0705c2addd217e427dbd4 WatchSource:0}: Error finding container 957dc4c2f263cfe8707cbe5d5f1d01c42da86893e6b0705c2addd217e427dbd4: Status 404 returned error can't find the container with id 957dc4c2f263cfe8707cbe5d5f1d01c42da86893e6b0705c2addd217e427dbd4
Sep 08 04:13:42.631408 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:42.631389    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"]
Sep 08 04:13:42.633117 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:13:42.633089    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae3b016_ca00_455d_b39d_5647a6b113d1.slice/crio-5843560e0a4c1d235826bcdf66eb36a86f250d2c5f7eb7b7f40e0fe24dbc2ba0 WatchSource:0}: Error finding container 5843560e0a4c1d235826bcdf66eb36a86f250d2c5f7eb7b7f40e0fe24dbc2ba0: Status 404 returned error can't find the container with id 5843560e0a4c1d235826bcdf66eb36a86f250d2c5f7eb7b7f40e0fe24dbc2ba0
Sep 08 04:13:43.160074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.160036    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"]
Sep 08 04:13:43.174173 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.174083    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"]
Sep 08 04:13:43.174324 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.174204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.181853 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.181250    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npqmj\""
Sep 08 04:13:43.181853 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.181419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 04:13:43.181853 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.181480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 04:13:43.181853 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.181679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:13:43.182474 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.182365    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f0271e-serving-cert\""
Sep 08 04:13:43.252051 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.252015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk" event={"ID":"fae3b016-ca00-455d-b39d-5647a6b113d1","Type":"ContainerStarted","Data":"5843560e0a4c1d235826bcdf66eb36a86f250d2c5f7eb7b7f40e0fe24dbc2ba0"}
Sep 08 04:13:43.254421 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.254390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx" event={"ID":"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6","Type":"ContainerStarted","Data":"957dc4c2f263cfe8707cbe5d5f1d01c42da86893e6b0705c2addd217e427dbd4"}
Sep 08 04:13:43.261407 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.261378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6szf\" (UniqueName: \"kubernetes.io/projected/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-kube-api-access-v6szf\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.261506 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.261427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.261506 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.261454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-metrics-certs\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.261506 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.261487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-openshift-service-ca\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.362172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6szf\" (UniqueName: \"kubernetes.io/projected/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-kube-api-access-v6szf\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.362221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.362252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-metrics-certs\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.362288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-openshift-service-ca\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.363076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-openshift-service-ca\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:43.363460    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-serving-cert: secret "maas-api-e2e-autores-f0271e-serving-cert" not found
Sep 08 04:13:43.363809 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:43.363520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls podName:d2984d1c-4d8d-4e24-9bcf-91668eb11f4b nodeName:}" failed. No retries permitted until 2026-09-08 04:13:43.863500799 +0000 UTC m=+804.221980354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls") pod "maas-api-e2e-autores-f0271e-64684d579c-tbpl6" (UID: "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b") : secret "maas-api-e2e-autores-f0271e-serving-cert" not found
Sep 08 04:13:43.368705 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.368661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-metrics-certs\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.381972 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.381949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6szf\" (UniqueName: \"kubernetes.io/projected/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-kube-api-access-v6szf\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.867553 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:43.867407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:43.867553 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:43.867541    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-serving-cert: secret "maas-api-e2e-autores-f0271e-serving-cert" not found
Sep 08 04:13:43.867812 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:13:43.867626    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls podName:d2984d1c-4d8d-4e24-9bcf-91668eb11f4b nodeName:}" failed. No retries permitted until 2026-09-08 04:13:44.867609929 +0000 UTC m=+805.226089466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls") pod "maas-api-e2e-autores-f0271e-64684d579c-tbpl6" (UID: "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b") : secret "maas-api-e2e-autores-f0271e-serving-cert" not found
Sep 08 04:13:44.671471 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:44.671431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:13:44.878024 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:44.877981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:44.880297 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:44.880266    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") pod \"maas-api-e2e-autores-f0271e-64684d579c-tbpl6\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:44.991406 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:44.991378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:45.267310 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.267271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx" event={"ID":"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6","Type":"ContainerStarted","Data":"d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375"}
Sep 08 04:13:45.267592 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.267570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:45.269842 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.269802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk" event={"ID":"fae3b016-ca00-455d-b39d-5647a6b113d1","Type":"ContainerStarted","Data":"7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1"}
Sep 08 04:13:45.270114 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.270052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:13:45.270710 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.270683    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"]
Sep 08 04:13:45.275245 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:13:45.275219    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2984d1c_4d8d_4e24_9bcf_91668eb11f4b.slice/crio-f09c49ebc6c51184bdfff5681195f2554a024ee889457cc96af3eb5eed21dd26 WatchSource:0}: Error finding container f09c49ebc6c51184bdfff5681195f2554a024ee889457cc96af3eb5eed21dd26: Status 404 returned error can't find the container with id f09c49ebc6c51184bdfff5681195f2554a024ee889457cc96af3eb5eed21dd26
Sep 08 04:13:45.285515 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.285475    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx" podStartSLOduration=1.9423528380000001 podStartE2EDuration="4.285462554s" totalImagesPullingTime="2.343109716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:45.284368081 +0000 UTC m=+805.642847643" watchObservedRunningTime="2026-09-08 04:13:45.285462554 +0000 UTC m=+805.643942157"
Sep 08 04:13:45.300429 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:45.300387    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk" podStartSLOduration=1.003406271 podStartE2EDuration="3.300370822s" totalImagesPullingTime="2.296964551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:45.298851285 +0000 UTC m=+805.657330847" watchObservedRunningTime="2026-09-08 04:13:45.300370822 +0000 UTC m=+805.658850381"
Sep 08 04:13:46.275476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:46.275438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6" event={"ID":"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b","Type":"ContainerStarted","Data":"f09c49ebc6c51184bdfff5681195f2554a024ee889457cc96af3eb5eed21dd26"}
Sep 08 04:13:47.281465 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:47.281428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6" event={"ID":"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b","Type":"ContainerStarted","Data":"2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088"}
Sep 08 04:13:47.281823 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:47.281561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:47.297486 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:47.297435    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6" podStartSLOduration=2.613548551 podStartE2EDuration="4.297418483s" totalImagesPullingTime="1.683869932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:47.296687529 +0000 UTC m=+807.655167090" watchObservedRunningTime="2026-09-08 04:13:47.297418483 +0000 UTC m=+807.655898040"
Sep 08 04:13:53.291395 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:53.291367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:13:56.277831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:56.277799    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:13:56.278172 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:13:56.277873    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:14:01.937383 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:01.937348    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"]
Sep 08 04:14:01.943462 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:01.943435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:01.946170 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:01.946144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cf5ae2-serving-cert\""
Sep 08 04:14:01.953071 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:01.953048    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"]
Sep 08 04:14:02.006462 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.006432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9935240-fd0e-4685-818e-b1dd07a5913e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.006567 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.006473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.006567 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.006537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-metrics-certs\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.006567 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.006555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6md\" (UniqueName: \"kubernetes.io/projected/c9935240-fd0e-4685-818e-b1dd07a5913e-kube-api-access-xk6md\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.107458 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.107431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-metrics-certs\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.107556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.107463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6md\" (UniqueName: \"kubernetes.io/projected/c9935240-fd0e-4685-818e-b1dd07a5913e-kube-api-access-xk6md\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.107556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.107497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9935240-fd0e-4685-818e-b1dd07a5913e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.107669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.107613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.107838 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:14:02.107820    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-serving-cert: secret "maas-api-e2e-inf-b-cf5ae2-serving-cert" not found
Sep 08 04:14:02.107924 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:14:02.107913    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls podName:c9935240-fd0e-4685-818e-b1dd07a5913e nodeName:}" failed. No retries permitted until 2026-09-08 04:14:02.607891976 +0000 UTC m=+822.966371517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls") pod "maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" (UID: "c9935240-fd0e-4685-818e-b1dd07a5913e") : secret "maas-api-e2e-inf-b-cf5ae2-serving-cert" not found
Sep 08 04:14:02.108180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.108158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9935240-fd0e-4685-818e-b1dd07a5913e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.109939 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.109916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-metrics-certs\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.118143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.118123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6md\" (UniqueName: \"kubernetes.io/projected/c9935240-fd0e-4685-818e-b1dd07a5913e-kube-api-access-xk6md\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.584006 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.583972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"]
Sep 08 04:14:02.587886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.587868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.592881 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.592857    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cf5ae2\""
Sep 08 04:14:02.599890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.599868    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"]
Sep 08 04:14:02.612848 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.612820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.615029 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.615003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.639943 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.639922    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"]
Sep 08 04:14:02.643644 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.643629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.645761 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.645733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cf5ae2-dockercfg-hgnjr\""
Sep 08 04:14:02.651054 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.651032    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"]
Sep 08 04:14:02.713198 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.713178    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbwx\" (UniqueName: \"kubernetes.io/projected/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-kube-api-access-6bbwx\") pod \"payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.713289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.713224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2nd\" (UniqueName: \"kubernetes.io/projected/d4c66763-c9ed-45fb-b024-07e242ebba69-kube-api-access-mc2nd\") pod \"payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.713289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.713251    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.713377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.713318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c66763-c9ed-45fb-b024-07e242ebba69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.814636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.814613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c66763-c9ed-45fb-b024-07e242ebba69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.814733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.814652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbwx\" (UniqueName: \"kubernetes.io/projected/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-kube-api-access-6bbwx\") pod \"payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.814733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.814694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2nd\" (UniqueName: \"kubernetes.io/projected/d4c66763-c9ed-45fb-b024-07e242ebba69-kube-api-access-mc2nd\") pod \"payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.814733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.814717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.815314 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.815290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c66763-c9ed-45fb-b024-07e242ebba69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.815379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.815363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.822573 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.822549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2nd\" (UniqueName: \"kubernetes.io/projected/d4c66763-c9ed-45fb-b024-07e242ebba69-kube-api-access-mc2nd\") pod \"payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:02.822664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.822635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbwx\" (UniqueName: \"kubernetes.io/projected/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-kube-api-access-6bbwx\") pod \"payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.860450 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.860366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:02.898293 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.898257    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:02.954331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:02.954281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:03.132908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.132888    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"]
Sep 08 04:14:03.134860 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:14:03.134828    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9935240_fd0e_4685_818e_b1dd07a5913e.slice/crio-2f3a19477b08beeec72fc5cfa3ef618d5afe9269f7f918591c805930e0373409 WatchSource:0}: Error finding container 2f3a19477b08beeec72fc5cfa3ef618d5afe9269f7f918591c805930e0373409: Status 404 returned error can't find the container with id 2f3a19477b08beeec72fc5cfa3ef618d5afe9269f7f918591c805930e0373409
Sep 08 04:14:03.179763 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.179725    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"]
Sep 08 04:14:03.180416 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:14:03.180392    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3469a1f2_461a_4f49_ba5b_d9d21dc6ccf5.slice/crio-4f0b77e90cf2a67e557de10d4fb7cce6f96539f9912a0976f6d2484731326c90 WatchSource:0}: Error finding container 4f0b77e90cf2a67e557de10d4fb7cce6f96539f9912a0976f6d2484731326c90: Status 404 returned error can't find the container with id 4f0b77e90cf2a67e557de10d4fb7cce6f96539f9912a0976f6d2484731326c90
Sep 08 04:14:03.237577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.237556    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"]
Sep 08 04:14:03.239519 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:14:03.239492    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c66763_c9ed_45fb_b024_07e242ebba69.slice/crio-f3e8bff856864be2126c0f54294bd63716a1f195f897199b4c8d4578161b9ab9 WatchSource:0}: Error finding container f3e8bff856864be2126c0f54294bd63716a1f195f897199b4c8d4578161b9ab9: Status 404 returned error can't find the container with id f3e8bff856864be2126c0f54294bd63716a1f195f897199b4c8d4578161b9ab9
Sep 08 04:14:03.351624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.351592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h" event={"ID":"d4c66763-c9ed-45fb-b024-07e242ebba69","Type":"ContainerStarted","Data":"ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4"}
Sep 08 04:14:03.351793 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.351682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h" event={"ID":"d4c66763-c9ed-45fb-b024-07e242ebba69","Type":"ContainerStarted","Data":"f3e8bff856864be2126c0f54294bd63716a1f195f897199b4c8d4578161b9ab9"}
Sep 08 04:14:03.351793 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.351714    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:03.352874 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.352843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" event={"ID":"c9935240-fd0e-4685-818e-b1dd07a5913e","Type":"ContainerStarted","Data":"2f3a19477b08beeec72fc5cfa3ef618d5afe9269f7f918591c805930e0373409"}
Sep 08 04:14:03.354633 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.354609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw" event={"ID":"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5","Type":"ContainerStarted","Data":"323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835"}
Sep 08 04:14:03.354723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.354658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw" event={"ID":"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5","Type":"ContainerStarted","Data":"4f0b77e90cf2a67e557de10d4fb7cce6f96539f9912a0976f6d2484731326c90"}
Sep 08 04:14:03.354723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.354708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:03.368646 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.368605    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h" podStartSLOduration=1.368591728 podStartE2EDuration="1.368591728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:03.366856275 +0000 UTC m=+823.725335837" watchObservedRunningTime="2026-09-08 04:14:03.368591728 +0000 UTC m=+823.727071288"
Sep 08 04:14:03.394871 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:03.394785    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw" podStartSLOduration=1.3947669569999999 podStartE2EDuration="1.394766957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:03.39383733 +0000 UTC m=+823.752316889" watchObservedRunningTime="2026-09-08 04:14:03.394766957 +0000 UTC m=+823.753246517"
Sep 08 04:14:04.360776 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:04.360716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" event={"ID":"c9935240-fd0e-4685-818e-b1dd07a5913e","Type":"ContainerStarted","Data":"67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c"}
Sep 08 04:14:04.361180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:04.360821    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:04.378390 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:04.378348    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" podStartSLOduration=2.9448257399999997 podStartE2EDuration="3.378335377s" totalImagesPullingTime="433.509637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:04.376618902 +0000 UTC m=+824.735098463" watchObservedRunningTime="2026-09-08 04:14:04.378335377 +0000 UTC m=+824.736814940"
Sep 08 04:14:05.542099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:05.542065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:14:10.370996 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:10.370962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:14:14.362911 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:14.362865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:14:14.363310 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:14.363054    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:14:20.141871 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.141841    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"]
Sep 08 04:14:20.144912 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.144885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.147539 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.147521    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-83699c91-kserve-self-signed-certs\""
Sep 08 04:14:20.153346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.153320    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"]
Sep 08 04:14:20.245764 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.245724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-home\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.245857 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.245772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-model-cache\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.245857 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.245823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1252-e432-4522-b7fd-928791a9ebf5-tls-certs\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.245931 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.245870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6rg\" (UniqueName: \"kubernetes.io/projected/641b1252-e432-4522-b7fd-928791a9ebf5-kube-api-access-lz6rg\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.245931 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.245916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-dshm\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.246052 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.246019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-tmp-dir\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.346601 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-home\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.346731 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-model-cache\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.346731 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1252-e432-4522-b7fd-928791a9ebf5-tls-certs\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.346731 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6rg\" (UniqueName: \"kubernetes.io/projected/641b1252-e432-4522-b7fd-928791a9ebf5-kube-api-access-lz6rg\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.346731 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-dshm\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.346989 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-tmp-dir\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.347055 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.346986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-home\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.347055 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.347010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-model-cache\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.347131 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.347119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-tmp-dir\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.348905 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.348886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-dshm\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.349016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.348995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1252-e432-4522-b7fd-928791a9ebf5-tls-certs\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.354058 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.354037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6rg\" (UniqueName: \"kubernetes.io/projected/641b1252-e432-4522-b7fd-928791a9ebf5-kube-api-access-lz6rg\") pod \"e2e-conflict-a-83699c91-kserve-76647469f-qzhbr\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") " pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.455112 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.455060    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:20.729678 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:20.729596    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"]
Sep 08 04:14:20.733084 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:14:20.733055    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641b1252_e432_4522_b7fd_928791a9ebf5.slice/crio-636aaef04512e19faca091d923bb4b108ae9aedd85a0d54a155f288210f46b54 WatchSource:0}: Error finding container 636aaef04512e19faca091d923bb4b108ae9aedd85a0d54a155f288210f46b54: Status 404 returned error can't find the container with id 636aaef04512e19faca091d923bb4b108ae9aedd85a0d54a155f288210f46b54
Sep 08 04:14:21.436908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:21.436872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr" event={"ID":"641b1252-e432-4522-b7fd-928791a9ebf5","Type":"ContainerStarted","Data":"cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8"}
Sep 08 04:14:21.437349 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:21.436915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr" event={"ID":"641b1252-e432-4522-b7fd-928791a9ebf5","Type":"ContainerStarted","Data":"636aaef04512e19faca091d923bb4b108ae9aedd85a0d54a155f288210f46b54"}
Sep 08 04:14:21.453090 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:21.453038    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr" podStartSLOduration=1.45301964 podStartE2EDuration="1.45301964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:21.451263547 +0000 UTC m=+841.809743107" watchObservedRunningTime="2026-09-08 04:14:21.45301964 +0000 UTC m=+841.811499199"
Sep 08 04:14:22.118790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.118762    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"]
Sep 08 04:14:22.118965 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.118942    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk" podUID="fae3b016-ca00-455d-b39d-5647a6b113d1" containerName="payload-processing" containerID="cri-o://7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1" gracePeriod=30
Sep 08 04:14:22.134117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.134090    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"]
Sep 08 04:14:22.134277 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.134255    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx" podUID="7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" containerName="payload-pre-processing" containerID="cri-o://d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375" gracePeriod=30
Sep 08 04:14:22.421987 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.421956    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:14:22.425873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.425854    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:14:22.444636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.444611    2583 generic.go:356] "Generic (PLEG): container finished" podID="fae3b016-ca00-455d-b39d-5647a6b113d1" containerID="7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1" exitCode=0
Sep 08 04:14:22.445021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.444658    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"
Sep 08 04:14:22.445021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.444691    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk" event={"ID":"fae3b016-ca00-455d-b39d-5647a6b113d1","Type":"ContainerDied","Data":"7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1"}
Sep 08 04:14:22.445021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.444756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk" event={"ID":"fae3b016-ca00-455d-b39d-5647a6b113d1","Type":"ContainerDied","Data":"5843560e0a4c1d235826bcdf66eb36a86f250d2c5f7eb7b7f40e0fe24dbc2ba0"}
Sep 08 04:14:22.445021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.444789    2583 scope.go:117] "RemoveContainer" containerID="7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1"
Sep 08 04:14:22.446241 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.446218    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" containerID="d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375" exitCode=0
Sep 08 04:14:22.446328 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.446256    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"
Sep 08 04:14:22.446328 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.446263    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx" event={"ID":"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6","Type":"ContainerDied","Data":"d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375"}
Sep 08 04:14:22.446328 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.446319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx" event={"ID":"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6","Type":"ContainerDied","Data":"957dc4c2f263cfe8707cbe5d5f1d01c42da86893e6b0705c2addd217e427dbd4"}
Sep 08 04:14:22.452673 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.452659    2583 scope.go:117] "RemoveContainer" containerID="7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1"
Sep 08 04:14:22.453281 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:14:22.453177    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1\": container with ID starting with 7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1 not found: ID does not exist" containerID="7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1"
Sep 08 04:14:22.453437 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.453251    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1"} err="failed to get container status \"7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1\": rpc error: code = NotFound desc = could not find container \"7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1\": container with ID starting with 7b2da61efdda17e18f82d9ca4a1e6246ae4f417fea914ff49eb21292abd834b1 not found: ID does not exist"
Sep 08 04:14:22.453537 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.453461    2583 scope.go:117] "RemoveContainer" containerID="d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375"
Sep 08 04:14:22.462011 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.461994    2583 scope.go:117] "RemoveContainer" containerID="d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375"
Sep 08 04:14:22.462272 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:14:22.462240    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375\": container with ID starting with d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375 not found: ID does not exist" containerID="d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375"
Sep 08 04:14:22.462357 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.462278    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375"} err="failed to get container status \"d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375\": rpc error: code = NotFound desc = could not find container \"d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375\": container with ID starting with d9d9c99cb8d9373eba94c9af8525d54aa7a821919cc24d64009f6a959598d375 not found: ID does not exist"
Sep 08 04:14:22.571050 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.571029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-plugins-config-volume\") pod \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") "
Sep 08 04:14:22.571134 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.571058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2252\" (UniqueName: \"kubernetes.io/projected/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-kube-api-access-w2252\") pod \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\" (UID: \"7d9a40f3-8ca0-4758-9fbb-e6f6246719c6\") "
Sep 08 04:14:22.571134 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.571099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf7sr\" (UniqueName: \"kubernetes.io/projected/fae3b016-ca00-455d-b39d-5647a6b113d1-kube-api-access-gf7sr\") pod \"fae3b016-ca00-455d-b39d-5647a6b113d1\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") "
Sep 08 04:14:22.571250 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.571134    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3b016-ca00-455d-b39d-5647a6b113d1-plugins-config-volume\") pod \"fae3b016-ca00-455d-b39d-5647a6b113d1\" (UID: \"fae3b016-ca00-455d-b39d-5647a6b113d1\") "
Sep 08 04:14:22.571541 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.571502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" (UID: "7d9a40f3-8ca0-4758-9fbb-e6f6246719c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:14:22.571636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.571543    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae3b016-ca00-455d-b39d-5647a6b113d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fae3b016-ca00-455d-b39d-5647a6b113d1" (UID: "fae3b016-ca00-455d-b39d-5647a6b113d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:14:22.573273 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.573252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-kube-api-access-w2252" (OuterVolumeSpecName: "kube-api-access-w2252") pod "7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" (UID: "7d9a40f3-8ca0-4758-9fbb-e6f6246719c6"). InnerVolumeSpecName "kube-api-access-w2252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:14:22.573340 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.573282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae3b016-ca00-455d-b39d-5647a6b113d1-kube-api-access-gf7sr" (OuterVolumeSpecName: "kube-api-access-gf7sr") pod "fae3b016-ca00-455d-b39d-5647a6b113d1" (UID: "fae3b016-ca00-455d-b39d-5647a6b113d1"). InnerVolumeSpecName "kube-api-access-gf7sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:14:22.671589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.671543    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2252\" (UniqueName: \"kubernetes.io/projected/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-kube-api-access-w2252\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:14:22.671589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.671564    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7sr\" (UniqueName: \"kubernetes.io/projected/fae3b016-ca00-455d-b39d-5647a6b113d1-kube-api-access-gf7sr\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:14:22.671589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.671575    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae3b016-ca00-455d-b39d-5647a6b113d1-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:14:22.671589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.671583    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:14:22.766833 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.766804    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"]
Sep 08 04:14:22.771025 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.771003    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d710ebd-7cb876d44f-vkpgk"]
Sep 08 04:14:22.779791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.779771    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"]
Sep 08 04:14:22.788389 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:22.788370    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d710ebd-54d8554bdb-lk5sx"]
Sep 08 04:14:24.159022 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:24.158991    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" path="/var/lib/kubelet/pods/7d9a40f3-8ca0-4758-9fbb-e6f6246719c6/volumes"
Sep 08 04:14:24.159360 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:24.159312    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae3b016-ca00-455d-b39d-5647a6b113d1" path="/var/lib/kubelet/pods/fae3b016-ca00-455d-b39d-5647a6b113d1/volumes"
Sep 08 04:14:26.244000 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:26.243963    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:14:30.455831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:30.455799    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:30.456185 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:30.455842    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:30.467954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:30.467927    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:30.499336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:30.499318    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:14:42.642964 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:14:42.642924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:15:00.140459 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140427    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"]
Sep 08 04:15:00.140827 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140769    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" containerName="payload-pre-processing"
Sep 08 04:15:00.140827 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140781    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" containerName="payload-pre-processing"
Sep 08 04:15:00.140827 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140791    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae3b016-ca00-455d-b39d-5647a6b113d1" containerName="payload-processing"
Sep 08 04:15:00.140827 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140797    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae3b016-ca00-455d-b39d-5647a6b113d1" containerName="payload-processing"
Sep 08 04:15:00.140957 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140853    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9a40f3-8ca0-4758-9fbb-e6f6246719c6" containerName="payload-pre-processing"
Sep 08 04:15:00.140957 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.140862    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae3b016-ca00-455d-b39d-5647a6b113d1" containerName="payload-processing"
Sep 08 04:15:00.143906 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.143891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"
Sep 08 04:15:00.146333 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.146311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\""
Sep 08 04:15:00.154985 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.154965    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"
Sep 08 04:15:00.161592 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.161567    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"]
Sep 08 04:15:00.423141 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.423114    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"]
Sep 08 04:15:00.425797 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:15:00.425772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaccbac6_9139_4e2f_bc8e_79ba3160c7c1.slice/crio-ad4c7341045f8d9881bf58ce65dacdc6889d26c1305156823a12db9bc065dfcf WatchSource:0}: Error finding container ad4c7341045f8d9881bf58ce65dacdc6889d26c1305156823a12db9bc065dfcf: Status 404 returned error can't find the container with id ad4c7341045f8d9881bf58ce65dacdc6889d26c1305156823a12db9bc065dfcf
Sep 08 04:15:00.616455 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:00.616428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn" event={"ID":"caccbac6-9139-4e2f-bc8e-79ba3160c7c1","Type":"ContainerStarted","Data":"ad4c7341045f8d9881bf58ce65dacdc6889d26c1305156823a12db9bc065dfcf"}
Sep 08 04:15:03.632348 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:03.632276    2583 generic.go:356] "Generic (PLEG): container finished" podID="caccbac6-9139-4e2f-bc8e-79ba3160c7c1" containerID="62e803dee80ee5fbdabb61ef0b1d01c0ff54b5e393d084b815d97a5c4b047aa6" exitCode=0
Sep 08 04:15:03.632640 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:03.632362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn" event={"ID":"caccbac6-9139-4e2f-bc8e-79ba3160c7c1","Type":"ContainerDied","Data":"62e803dee80ee5fbdabb61ef0b1d01c0ff54b5e393d084b815d97a5c4b047aa6"}
Sep 08 04:15:04.765373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:04.765348    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"
Sep 08 04:15:05.643419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:05.643380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn" event={"ID":"caccbac6-9139-4e2f-bc8e-79ba3160c7c1","Type":"ContainerDied","Data":"ad4c7341045f8d9881bf58ce65dacdc6889d26c1305156823a12db9bc065dfcf"}
Sep 08 04:15:05.643419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:05.643403    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"
Sep 08 04:15:05.643640 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:05.643466    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4c7341045f8d9881bf58ce65dacdc6889d26c1305156823a12db9bc065dfcf"
Sep 08 04:15:06.747654 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:06.747618    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:15:20.123529 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:20.123503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:15:20.123998 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:20.123906    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:15:25.038778 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.038721    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"]
Sep 08 04:15:25.039259 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.039051    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caccbac6-9139-4e2f-bc8e-79ba3160c7c1" containerName="cleanup"
Sep 08 04:15:25.039259 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.039062    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="caccbac6-9139-4e2f-bc8e-79ba3160c7c1" containerName="cleanup"
Sep 08 04:15:25.039259 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.039121    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="caccbac6-9139-4e2f-bc8e-79ba3160c7c1" containerName="cleanup"
Sep 08 04:15:25.043299 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.043277    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.045610 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.045588    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f0271e\"/\"openshift-service-ca.crt\""
Sep 08 04:15:25.046652 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.046574    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f0271e\"/\"kube-root-ca.crt\""
Sep 08 04:15:25.046652 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.046610    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f0271e\"/\"default-dockercfg-5qtx9\""
Sep 08 04:15:25.046956 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.046934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f0271e\"/\"autores-model-f0271e-kserve-self-signed-certs\""
Sep 08 04:15:25.048830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.048807    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"]
Sep 08 04:15:25.114792 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.114769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tmp-dir\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.114889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.114823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjvj\" (UniqueName: \"kubernetes.io/projected/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-kube-api-access-2wjvj\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.114889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.114883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-home\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.114955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.114900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tls-certs\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.114955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.114917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-model-cache\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.114955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.114933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-dshm\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215576 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-home\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tls-certs\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-model-cache\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215614    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-dshm\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tmp-dir\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215846 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjvj\" (UniqueName: \"kubernetes.io/projected/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-kube-api-access-2wjvj\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.215960 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.215935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-model-cache\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.216072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.216036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-home\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.216072 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.216062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tmp-dir\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.217893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.217872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-dshm\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.218084 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.218069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tls-certs\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.223570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.223546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjvj\" (UniqueName: \"kubernetes.io/projected/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-kube-api-access-2wjvj\") pod \"autores-model-f0271e-kserve-64ccb75f9f-k9ch9\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") " pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.352888 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.352815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:25.620259 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.620234    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"]
Sep 08 04:15:25.622423 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:15:25.622397    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6592bd27_ed5b_4a5a_b6c6_2e87c594ed44.slice/crio-3a25d56a6af21e8f8a8a44a0469a8949e2e041a12b67ffd1719b9678f1768117 WatchSource:0}: Error finding container 3a25d56a6af21e8f8a8a44a0469a8949e2e041a12b67ffd1719b9678f1768117: Status 404 returned error can't find the container with id 3a25d56a6af21e8f8a8a44a0469a8949e2e041a12b67ffd1719b9678f1768117
Sep 08 04:15:25.736426 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.736397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9" event={"ID":"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44","Type":"ContainerStarted","Data":"6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8"}
Sep 08 04:15:25.736589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.736439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9" event={"ID":"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44","Type":"ContainerStarted","Data":"3a25d56a6af21e8f8a8a44a0469a8949e2e041a12b67ffd1719b9678f1768117"}
Sep 08 04:15:25.753384 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:25.753328    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9" podStartSLOduration=0.753312547 podStartE2EDuration="753.312547ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:15:25.752060087 +0000 UTC m=+906.110539646" watchObservedRunningTime="2026-09-08 04:15:25.753312547 +0000 UTC m=+906.111792106"
Sep 08 04:15:26.642116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:26.642081    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:15:35.353562 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:35.353512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:35.354046 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:35.353577    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:35.367431 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:35.367401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:35.791152 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:35.791120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:15:43.355464 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:15:43.355421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:16:03.146196 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:03.146164    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:16:04.343723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.343689    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"]
Sep 08 04:16:04.347082 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.347060    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.349354 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.349330    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-83699c91-kserve-self-signed-certs\""
Sep 08 04:16:04.357605 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.357584    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"]
Sep 08 04:16:04.439266 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.439239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-model-cache\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.439373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.439284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-dshm\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.439373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.439309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tmp-dir\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.439503 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.439412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72zc9\" (UniqueName: \"kubernetes.io/projected/54dd080a-aaef-4cbb-b7b4-d753890ebf68-kube-api-access-72zc9\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.439503 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.439442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tls-certs\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.439503 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.439461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-home\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.540554 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.540524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72zc9\" (UniqueName: \"kubernetes.io/projected/54dd080a-aaef-4cbb-b7b4-d753890ebf68-kube-api-access-72zc9\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.540662 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.540565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tls-certs\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.540662 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.540589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-home\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.540662 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.540638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-model-cache\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.540841 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.540681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-dshm\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.540841 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.540696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tmp-dir\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.541089 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.541066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-home\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.541152 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.541096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tmp-dir\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.541152 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.541124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-model-cache\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.542822 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.542802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-dshm\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.542933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.542916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tls-certs\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.547626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.547605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72zc9\" (UniqueName: \"kubernetes.io/projected/54dd080a-aaef-4cbb-b7b4-d753890ebf68-kube-api-access-72zc9\") pod \"e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") " pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.657607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.657551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:04.927470 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:04.927445    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"]
Sep 08 04:16:04.929960 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:16:04.929936    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54dd080a_aaef_4cbb_b7b4_d753890ebf68.slice/crio-995a40c74579fcc40b46f1b555e4892d82ad6fa38473de5957669d1a22c9ed5e WatchSource:0}: Error finding container 995a40c74579fcc40b46f1b555e4892d82ad6fa38473de5957669d1a22c9ed5e: Status 404 returned error can't find the container with id 995a40c74579fcc40b46f1b555e4892d82ad6fa38473de5957669d1a22c9ed5e
Sep 08 04:16:05.922379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:05.922333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5" event={"ID":"54dd080a-aaef-4cbb-b7b4-d753890ebf68","Type":"ContainerStarted","Data":"704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa"}
Sep 08 04:16:05.922379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:05.922376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5" event={"ID":"54dd080a-aaef-4cbb-b7b4-d753890ebf68","Type":"ContainerStarted","Data":"995a40c74579fcc40b46f1b555e4892d82ad6fa38473de5957669d1a22c9ed5e"}
Sep 08 04:16:05.938929 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:05.938875    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5" podStartSLOduration=1.938856557 podStartE2EDuration="1.938856557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:16:05.938690862 +0000 UTC m=+946.297170422" watchObservedRunningTime="2026-09-08 04:16:05.938856557 +0000 UTC m=+946.297336119"
Sep 08 04:16:11.958086 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:11.958053    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"]
Sep 08 04:16:11.958547 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:11.958312    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6" podUID="d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" containerName="maas-api" containerID="cri-o://2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088" gracePeriod=30
Sep 08 04:16:12.205385 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.205360    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:16:12.313064 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.313036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-openshift-service-ca\") pod \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") "
Sep 08 04:16:12.313203 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.313071    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") pod \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") "
Sep 08 04:16:12.313203 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.313091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-metrics-certs\") pod \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") "
Sep 08 04:16:12.313203 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.313122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6szf\" (UniqueName: \"kubernetes.io/projected/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-kube-api-access-v6szf\") pod \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\" (UID: \"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b\") "
Sep 08 04:16:12.313444 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.313418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" (UID: "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:16:12.315261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.315240    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" (UID: "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:16:12.315367 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.315288    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-kube-api-access-v6szf" (OuterVolumeSpecName: "kube-api-access-v6szf") pod "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" (UID: "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b"). InnerVolumeSpecName "kube-api-access-v6szf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:16:12.315367 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.315308    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" (UID: "d2984d1c-4d8d-4e24-9bcf-91668eb11f4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:16:12.414241 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.414209    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6szf\" (UniqueName: \"kubernetes.io/projected/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-kube-api-access-v6szf\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:12.414241 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.414235    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:12.414412 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.414251    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:12.414412 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.414263    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:12.958512 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.958480    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" containerID="2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088" exitCode=0
Sep 08 04:16:12.958886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.958522    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"
Sep 08 04:16:12.958886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.958518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6" event={"ID":"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b","Type":"ContainerDied","Data":"2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088"}
Sep 08 04:16:12.958886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.958635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6" event={"ID":"d2984d1c-4d8d-4e24-9bcf-91668eb11f4b","Type":"ContainerDied","Data":"f09c49ebc6c51184bdfff5681195f2554a024ee889457cc96af3eb5eed21dd26"}
Sep 08 04:16:12.958886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.958660    2583 scope.go:117] "RemoveContainer" containerID="2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088"
Sep 08 04:16:12.968047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.968031    2583 scope.go:117] "RemoveContainer" containerID="2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088"
Sep 08 04:16:12.968302 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:16:12.968283    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088\": container with ID starting with 2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088 not found: ID does not exist" containerID="2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088"
Sep 08 04:16:12.968355 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.968309    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088"} err="failed to get container status \"2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088\": rpc error: code = NotFound desc = could not find container \"2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088\": container with ID starting with 2203e57be53d99c829fc830d63c70e5a610cdd58d6cffb2b48bd43c9e2596088 not found: ID does not exist"
Sep 08 04:16:12.980015 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.979995    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"]
Sep 08 04:16:12.985685 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:12.985664    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0271e-64684d579c-tbpl6"]
Sep 08 04:16:14.158809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:14.158768    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" path="/var/lib/kubelet/pods/d2984d1c-4d8d-4e24-9bcf-91668eb11f4b/volumes"
Sep 08 04:16:14.657813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:14.657782    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:14.657997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:14.657830    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:14.670016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:14.669993    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:14.978998 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:14.978930    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:16:16.645042 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:16.645004    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:16:17.526988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:17.526960    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"]
Sep 08 04:16:32.546499 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.546457    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9" podUID="6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" containerName="main" containerID="cri-o://6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8" gracePeriod=45
Sep 08 04:16:32.784632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.784611    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:16:32.861218 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861151    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-home\") pod \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") "
Sep 08 04:16:32.861218 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tmp-dir\") pod \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") "
Sep 08 04:16:32.861386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-model-cache\") pod \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") "
Sep 08 04:16:32.861386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861312    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-dshm\") pod \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") "
Sep 08 04:16:32.861386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wjvj\" (UniqueName: \"kubernetes.io/projected/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-kube-api-access-2wjvj\") pod \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") "
Sep 08 04:16:32.861386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tls-certs\") pod \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\" (UID: \"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44\") "
Sep 08 04:16:32.861580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-home" (OuterVolumeSpecName: "home") pod "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" (UID: "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:16:32.861580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-model-cache" (OuterVolumeSpecName: "model-cache") pod "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" (UID: "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:16:32.861580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" (UID: "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:16:32.861728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861657    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:32.861728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861692    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:32.861728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.861703    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:32.863589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.863558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-dshm" (OuterVolumeSpecName: "dshm") pod "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" (UID: "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:16:32.863714 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.863631    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" (UID: "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:16:32.863714 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.863679    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-kube-api-access-2wjvj" (OuterVolumeSpecName: "kube-api-access-2wjvj") pod "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" (UID: "6592bd27-ed5b-4a5a-b6c6-2e87c594ed44"). InnerVolumeSpecName "kube-api-access-2wjvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:16:32.962710 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.962687    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:32.962834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.962710    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjvj\" (UniqueName: \"kubernetes.io/projected/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-kube-api-access-2wjvj\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:32.962834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:32.962726    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:16:33.046123 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.046099    2583 generic.go:356] "Generic (PLEG): container finished" podID="6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" containerID="6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8" exitCode=0
Sep 08 04:16:33.046228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.046151    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"
Sep 08 04:16:33.046228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.046158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9" event={"ID":"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44","Type":"ContainerDied","Data":"6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8"}
Sep 08 04:16:33.046228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.046189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9" event={"ID":"6592bd27-ed5b-4a5a-b6c6-2e87c594ed44","Type":"ContainerDied","Data":"3a25d56a6af21e8f8a8a44a0469a8949e2e041a12b67ffd1719b9678f1768117"}
Sep 08 04:16:33.046228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.046205    2583 scope.go:117] "RemoveContainer" containerID="6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8"
Sep 08 04:16:33.055030 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.055015    2583 scope.go:117] "RemoveContainer" containerID="6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8"
Sep 08 04:16:33.055301 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:16:33.055280    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8\": container with ID starting with 6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8 not found: ID does not exist" containerID="6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8"
Sep 08 04:16:33.055341 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.055309    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8"} err="failed to get container status \"6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8\": rpc error: code = NotFound desc = could not find container \"6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8\": container with ID starting with 6f22584b3bd3c4f52a96827871a568daae63336ad1a855da7df29b7271325eb8 not found: ID does not exist"
Sep 08 04:16:33.072290 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.072270    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"]
Sep 08 04:16:33.076137 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.076116    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f0271e/autores-model-f0271e-kserve-64ccb75f9f-k9ch9"]
Sep 08 04:16:33.638452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:33.638426    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:16:34.158895 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:34.158863    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" path="/var/lib/kubelet/pods/6592bd27-ed5b-4a5a-b6c6-2e87c594ed44/volumes"
Sep 08 04:16:40.448042 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448007    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"]
Sep 08 04:16:40.448452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448317    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" containerName="maas-api"
Sep 08 04:16:40.448452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448327    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" containerName="maas-api"
Sep 08 04:16:40.448452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448338    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" containerName="main"
Sep 08 04:16:40.448452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" containerName="main"
Sep 08 04:16:40.448452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448400    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2984d1c-4d8d-4e24-9bcf-91668eb11f4b" containerName="maas-api"
Sep 08 04:16:40.448452 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.448406    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6592bd27-ed5b-4a5a-b6c6-2e87c594ed44" containerName="main"
Sep 08 04:16:40.452639 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.452620    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.456052 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.456028    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ff0f3e\"/\"kube-root-ca.crt\""
Sep 08 04:16:40.456174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.456067    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ff0f3e\"/\"openshift-service-ca.crt\""
Sep 08 04:16:40.456174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.456131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ff0f3e\"/\"default-dockercfg-9ct56\""
Sep 08 04:16:40.456283 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.456274    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ff0f3e\"/\"test-model-ff0f3e-kserve-self-signed-certs\""
Sep 08 04:16:40.462480 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.462454    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"]
Sep 08 04:16:40.519085 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.519055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-dshm\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.519180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.519086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58270842-164a-40cd-8dce-5413f9dc5e82-tls-certs\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.519180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.519115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2v8s\" (UniqueName: \"kubernetes.io/projected/58270842-164a-40cd-8dce-5413f9dc5e82-kube-api-access-g2v8s\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.519180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.519164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-tmp-dir\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.519282 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.519244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-model-cache\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.519282 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.519266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-home\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620508 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-dshm\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58270842-164a-40cd-8dce-5413f9dc5e82-tls-certs\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2v8s\" (UniqueName: \"kubernetes.io/projected/58270842-164a-40cd-8dce-5413f9dc5e82-kube-api-access-g2v8s\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-tmp-dir\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620708 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-model-cache\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620708 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-home\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-home\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.620992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.620967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-tmp-dir\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.621111 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.621014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-model-cache\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.622653 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.622633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-dshm\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.623056 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.623036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58270842-164a-40cd-8dce-5413f9dc5e82-tls-certs\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.628550 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.628529    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2v8s\" (UniqueName: \"kubernetes.io/projected/58270842-164a-40cd-8dce-5413f9dc5e82-kube-api-access-g2v8s\") pod \"test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") " pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:40.761943 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:40.761922    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:41.031334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:41.031264    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"]
Sep 08 04:16:41.034182 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:16:41.034157    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58270842_164a_40cd_8dce_5413f9dc5e82.slice/crio-6ddd95206f2f5244f25b12d0c53d6aacb4619f5300a91589d5889724f7f88256 WatchSource:0}: Error finding container 6ddd95206f2f5244f25b12d0c53d6aacb4619f5300a91589d5889724f7f88256: Status 404 returned error can't find the container with id 6ddd95206f2f5244f25b12d0c53d6aacb4619f5300a91589d5889724f7f88256
Sep 08 04:16:41.083778 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:41.083753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9" event={"ID":"58270842-164a-40cd-8dce-5413f9dc5e82","Type":"ContainerStarted","Data":"6ddd95206f2f5244f25b12d0c53d6aacb4619f5300a91589d5889724f7f88256"}
Sep 08 04:16:42.089590 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:42.089554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9" event={"ID":"58270842-164a-40cd-8dce-5413f9dc5e82","Type":"ContainerStarted","Data":"f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527"}
Sep 08 04:16:42.105089 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:42.105042    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9" podStartSLOduration=2.105027197 podStartE2EDuration="2.105027197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:16:42.104795449 +0000 UTC m=+982.463275010" watchObservedRunningTime="2026-09-08 04:16:42.105027197 +0000 UTC m=+982.463506757"
Sep 08 04:16:44.781464 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:44.781389    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:16:45.172648 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:45.172574    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"]
Sep 08 04:16:45.833470 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:45.833439    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"]
Sep 08 04:16:50.762429 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:50.762397    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:50.762838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:50.762445    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:50.774698 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:50.774675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:16:51.137358 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:16:51.137290    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:17:00.191935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.191896    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr" podUID="641b1252-e432-4522-b7fd-928791a9ebf5" containerName="main" containerID="cri-o://cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8" gracePeriod=45
Sep 08 04:17:00.440071 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.440046    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:17:00.572682 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572654    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1252-e432-4522-b7fd-928791a9ebf5-tls-certs\") pod \"641b1252-e432-4522-b7fd-928791a9ebf5\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") "
Sep 08 04:17:00.572810 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-model-cache\") pod \"641b1252-e432-4522-b7fd-928791a9ebf5\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") "
Sep 08 04:17:00.572810 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-tmp-dir\") pod \"641b1252-e432-4522-b7fd-928791a9ebf5\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") "
Sep 08 04:17:00.572810 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-dshm\") pod \"641b1252-e432-4522-b7fd-928791a9ebf5\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") "
Sep 08 04:17:00.572810 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz6rg\" (UniqueName: \"kubernetes.io/projected/641b1252-e432-4522-b7fd-928791a9ebf5-kube-api-access-lz6rg\") pod \"641b1252-e432-4522-b7fd-928791a9ebf5\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") "
Sep 08 04:17:00.573002 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572867    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-home\") pod \"641b1252-e432-4522-b7fd-928791a9ebf5\" (UID: \"641b1252-e432-4522-b7fd-928791a9ebf5\") "
Sep 08 04:17:00.573002 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.572987    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "641b1252-e432-4522-b7fd-928791a9ebf5" (UID: "641b1252-e432-4522-b7fd-928791a9ebf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:00.573111 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.573016    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "641b1252-e432-4522-b7fd-928791a9ebf5" (UID: "641b1252-e432-4522-b7fd-928791a9ebf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:00.573111 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.573100    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:00.573209 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.573114    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:00.573209 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.573196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-home" (OuterVolumeSpecName: "home") pod "641b1252-e432-4522-b7fd-928791a9ebf5" (UID: "641b1252-e432-4522-b7fd-928791a9ebf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:00.574823 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.574801    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641b1252-e432-4522-b7fd-928791a9ebf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "641b1252-e432-4522-b7fd-928791a9ebf5" (UID: "641b1252-e432-4522-b7fd-928791a9ebf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:17:00.574886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.574875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-dshm" (OuterVolumeSpecName: "dshm") pod "641b1252-e432-4522-b7fd-928791a9ebf5" (UID: "641b1252-e432-4522-b7fd-928791a9ebf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:00.574918 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.574883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641b1252-e432-4522-b7fd-928791a9ebf5-kube-api-access-lz6rg" (OuterVolumeSpecName: "kube-api-access-lz6rg") pod "641b1252-e432-4522-b7fd-928791a9ebf5" (UID: "641b1252-e432-4522-b7fd-928791a9ebf5"). InnerVolumeSpecName "kube-api-access-lz6rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:17:00.673996 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.673970    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:00.673996 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.673993    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1252-e432-4522-b7fd-928791a9ebf5-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:00.674116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.674008    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1252-e432-4522-b7fd-928791a9ebf5-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:00.674116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.674020    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz6rg\" (UniqueName: \"kubernetes.io/projected/641b1252-e432-4522-b7fd-928791a9ebf5-kube-api-access-lz6rg\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:00.852682 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:00.852601    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5" podUID="54dd080a-aaef-4cbb-b7b4-d753890ebf68" containerName="main" containerID="cri-o://704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa" gracePeriod=45
Sep 08 04:17:01.086319 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.086294    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:17:01.171418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.171343    2583 generic.go:356] "Generic (PLEG): container finished" podID="54dd080a-aaef-4cbb-b7b4-d753890ebf68" containerID="704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa" exitCode=0
Sep 08 04:17:01.171418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.171370    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"
Sep 08 04:17:01.171613 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.171421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5" event={"ID":"54dd080a-aaef-4cbb-b7b4-d753890ebf68","Type":"ContainerDied","Data":"704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa"}
Sep 08 04:17:01.171613 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.171463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5" event={"ID":"54dd080a-aaef-4cbb-b7b4-d753890ebf68","Type":"ContainerDied","Data":"995a40c74579fcc40b46f1b555e4892d82ad6fa38473de5957669d1a22c9ed5e"}
Sep 08 04:17:01.171613 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.171489    2583 scope.go:117] "RemoveContainer" containerID="704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa"
Sep 08 04:17:01.172997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.172973    2583 generic.go:356] "Generic (PLEG): container finished" podID="641b1252-e432-4522-b7fd-928791a9ebf5" containerID="cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8" exitCode=0
Sep 08 04:17:01.173091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.173026    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"
Sep 08 04:17:01.173091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.173054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr" event={"ID":"641b1252-e432-4522-b7fd-928791a9ebf5","Type":"ContainerDied","Data":"cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8"}
Sep 08 04:17:01.173091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.173079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr" event={"ID":"641b1252-e432-4522-b7fd-928791a9ebf5","Type":"ContainerDied","Data":"636aaef04512e19faca091d923bb4b108ae9aedd85a0d54a155f288210f46b54"}
Sep 08 04:17:01.177976 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.177954    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-home\") pod \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") "
Sep 08 04:17:01.178086 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tls-certs\") pod \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") "
Sep 08 04:17:01.178086 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178048    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72zc9\" (UniqueName: \"kubernetes.io/projected/54dd080a-aaef-4cbb-b7b4-d753890ebf68-kube-api-access-72zc9\") pod \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") "
Sep 08 04:17:01.178086 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178075    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-dshm\") pod \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") "
Sep 08 04:17:01.178248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-model-cache\") pod \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") "
Sep 08 04:17:01.178248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tmp-dir\") pod \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\" (UID: \"54dd080a-aaef-4cbb-b7b4-d753890ebf68\") "
Sep 08 04:17:01.178551 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-home" (OuterVolumeSpecName: "home") pod "54dd080a-aaef-4cbb-b7b4-d753890ebf68" (UID: "54dd080a-aaef-4cbb-b7b4-d753890ebf68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:01.178551 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178520    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-model-cache" (OuterVolumeSpecName: "model-cache") pod "54dd080a-aaef-4cbb-b7b4-d753890ebf68" (UID: "54dd080a-aaef-4cbb-b7b4-d753890ebf68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:01.178690 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.178545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54dd080a-aaef-4cbb-b7b4-d753890ebf68" (UID: "54dd080a-aaef-4cbb-b7b4-d753890ebf68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:01.180141 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.180118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-dshm" (OuterVolumeSpecName: "dshm") pod "54dd080a-aaef-4cbb-b7b4-d753890ebf68" (UID: "54dd080a-aaef-4cbb-b7b4-d753890ebf68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:17:01.180499 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.180472    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54dd080a-aaef-4cbb-b7b4-d753890ebf68" (UID: "54dd080a-aaef-4cbb-b7b4-d753890ebf68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:17:01.180557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.180497    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dd080a-aaef-4cbb-b7b4-d753890ebf68-kube-api-access-72zc9" (OuterVolumeSpecName: "kube-api-access-72zc9") pod "54dd080a-aaef-4cbb-b7b4-d753890ebf68" (UID: "54dd080a-aaef-4cbb-b7b4-d753890ebf68"). InnerVolumeSpecName "kube-api-access-72zc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:17:01.181449 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.181435    2583 scope.go:117] "RemoveContainer" containerID="704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa"
Sep 08 04:17:01.181761 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:17:01.181723    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa\": container with ID starting with 704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa not found: ID does not exist" containerID="704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa"
Sep 08 04:17:01.181838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.181770    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa"} err="failed to get container status \"704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa\": rpc error: code = NotFound desc = could not find container \"704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa\": container with ID starting with 704335cb1e3029d4c1f9f12be07d211f92d285c0a0682d66ada0c218a41b41aa not found: ID does not exist"
Sep 08 04:17:01.181838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.181794    2583 scope.go:117] "RemoveContainer" containerID="cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8"
Sep 08 04:17:01.189261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.189244    2583 scope.go:117] "RemoveContainer" containerID="cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8"
Sep 08 04:17:01.189514 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:17:01.189496    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8\": container with ID starting with cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8 not found: ID does not exist" containerID="cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8"
Sep 08 04:17:01.189566 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.189519    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8"} err="failed to get container status \"cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8\": rpc error: code = NotFound desc = could not find container \"cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8\": container with ID starting with cb876142368d37d9452fcad0f715cb4119b1a532af88c94ced43a58134431ae8 not found: ID does not exist"
Sep 08 04:17:01.195363 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.195343    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"]
Sep 08 04:17:01.198899 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.198877    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-83699c91-kserve-76647469f-qzhbr"]
Sep 08 04:17:01.279088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.279068    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:01.279166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.279090    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72zc9\" (UniqueName: \"kubernetes.io/projected/54dd080a-aaef-4cbb-b7b4-d753890ebf68-kube-api-access-72zc9\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:01.279166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.279102    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:01.279166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.279111    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:01.279166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.279120    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:01.279166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.279127    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dd080a-aaef-4cbb-b7b4-d753890ebf68-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:17:01.499816 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.499793    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"]
Sep 08 04:17:01.510839 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:01.510820    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-83699c91-kserve-54ddd65cc7-j5ms5"]
Sep 08 04:17:02.058407 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:02.058372    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:17:02.159938 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:02.159896    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54dd080a-aaef-4cbb-b7b4-d753890ebf68" path="/var/lib/kubelet/pods/54dd080a-aaef-4cbb-b7b4-d753890ebf68/volumes"
Sep 08 04:17:02.160308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:02.160286    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641b1252-e432-4522-b7fd-928791a9ebf5" path="/var/lib/kubelet/pods/641b1252-e432-4522-b7fd-928791a9ebf5/volumes"
Sep 08 04:17:16.545205 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545170    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"]
Sep 08 04:17:16.545671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545484    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641b1252-e432-4522-b7fd-928791a9ebf5" containerName="main"
Sep 08 04:17:16.545671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545495    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="641b1252-e432-4522-b7fd-928791a9ebf5" containerName="main"
Sep 08 04:17:16.545671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545512    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dd080a-aaef-4cbb-b7b4-d753890ebf68" containerName="main"
Sep 08 04:17:16.545671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545518    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dd080a-aaef-4cbb-b7b4-d753890ebf68" containerName="main"
Sep 08 04:17:16.545671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545590    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dd080a-aaef-4cbb-b7b4-d753890ebf68" containerName="main"
Sep 08 04:17:16.545671 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.545598    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="641b1252-e432-4522-b7fd-928791a9ebf5" containerName="main"
Sep 08 04:17:16.550170 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.550151    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.553392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.553372    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf5ae2\"/\"kube-root-ca.crt\""
Sep 08 04:17:16.553392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.553379    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf5ae2\"/\"openshift-service-ca.crt\""
Sep 08 04:17:16.553556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.553465    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf5ae2\"/\"test-model-cf5ae2-kserve-self-signed-certs\""
Sep 08 04:17:16.553556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.553520    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf5ae2\"/\"default-dockercfg-4sk8w\""
Sep 08 04:17:16.558762 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.558713    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"]
Sep 08 04:17:16.584517 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.584488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k68m7\" (UniqueName: \"kubernetes.io/projected/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-kube-api-access-k68m7\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.584612 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.584526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tmp-dir\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.584612 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.584548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-dshm\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.584612 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.584589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-home\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.584731 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.584619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tls-certs\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.584731 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.584675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-model-cache\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.685800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.685777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-home\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.685908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.685813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tls-certs\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.685908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.685850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-model-cache\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.685908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.685899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k68m7\" (UniqueName: \"kubernetes.io/projected/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-kube-api-access-k68m7\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.686061 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.685933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tmp-dir\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.686061 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.685970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-dshm\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.686164 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.686144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-home\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.686222 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.686186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-model-cache\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.686276 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.686246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tmp-dir\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.688082 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.688062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-dshm\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.688152 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.688088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tls-certs\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.695888 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.695869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k68m7\" (UniqueName: \"kubernetes.io/projected/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-kube-api-access-k68m7\") pod \"test-model-cf5ae2-kserve-9db94c749-kjtgm\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") " pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:16.859968 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:16.859912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:17.132509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:17.132486    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"]
Sep 08 04:17:17.134372 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:17:17.134349    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8c32540_f7c1_4926_b58f_d57b1dd7fc50.slice/crio-da17f4250ae86f5dd906047cef452a449f09d003f070806e7b9d4749fd68c673 WatchSource:0}: Error finding container da17f4250ae86f5dd906047cef452a449f09d003f070806e7b9d4749fd68c673: Status 404 returned error can't find the container with id da17f4250ae86f5dd906047cef452a449f09d003f070806e7b9d4749fd68c673
Sep 08 04:17:17.242689 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:17.242650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm" event={"ID":"e8c32540-f7c1-4926-b58f-d57b1dd7fc50","Type":"ContainerStarted","Data":"31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721"}
Sep 08 04:17:17.242873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:17.242699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm" event={"ID":"e8c32540-f7c1-4926-b58f-d57b1dd7fc50","Type":"ContainerStarted","Data":"da17f4250ae86f5dd906047cef452a449f09d003f070806e7b9d4749fd68c673"}
Sep 08 04:17:17.263447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:17.263400    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm" podStartSLOduration=1.263387568 podStartE2EDuration="1.263387568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:17:17.260988791 +0000 UTC m=+1017.619468341" watchObservedRunningTime="2026-09-08 04:17:17.263387568 +0000 UTC m=+1017.621867127"
Sep 08 04:17:20.253876 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:20.253849    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:17:26.860158 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:26.860114    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:26.860585 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:26.860180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:26.872281 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:26.872253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:27.297413 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:27.297379    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:17:34.850469 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:34.850434    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:17:38.149228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.149186    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"]
Sep 08 04:17:38.152604 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.152581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.156252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.156231    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cb3759\"/\"kube-root-ca.crt\""
Sep 08 04:17:38.156377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.156260    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-cb3759\"/\"openshift-service-ca.crt\""
Sep 08 04:17:38.156377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.156231    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cb3759\"/\"default-dockercfg-ncrnm\""
Sep 08 04:17:38.156377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.156231    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-cb3759\"/\"explicit-model-cb3759-kserve-self-signed-certs\""
Sep 08 04:17:38.168811 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.168787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"]
Sep 08 04:17:38.239599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.239565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37987e4d-135f-4185-aee5-2bab62703d66-tls-certs\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.239790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.239614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-dshm\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.239790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.239639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-model-cache\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.239790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.239663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-tmp-dir\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.239790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.239690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-home\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.239790 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.239705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8cx\" (UniqueName: \"kubernetes.io/projected/37987e4d-135f-4185-aee5-2bab62703d66-kube-api-access-xc8cx\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.340809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.340778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37987e4d-135f-4185-aee5-2bab62703d66-tls-certs\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.340983 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.340865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-dshm\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.340983 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.340893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-model-cache\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.340983 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.340916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-tmp-dir\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.340983 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.340949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-home\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.340983 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.340973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8cx\" (UniqueName: \"kubernetes.io/projected/37987e4d-135f-4185-aee5-2bab62703d66-kube-api-access-xc8cx\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.341389 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.341349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-model-cache\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.341513 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.341386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-tmp-dir\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.341513 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.341399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-home\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.343215 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.343195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37987e4d-135f-4185-aee5-2bab62703d66-tls-certs\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.343297 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.343246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-dshm\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.350889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.350869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8cx\" (UniqueName: \"kubernetes.io/projected/37987e4d-135f-4185-aee5-2bab62703d66-kube-api-access-xc8cx\") pod \"explicit-model-cb3759-kserve-786b7bf55f-99txd\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") " pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.462075 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.461996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:38.732802 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:38.732774    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"]
Sep 08 04:17:38.734336 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:17:38.734300    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37987e4d_135f_4185_aee5_2bab62703d66.slice/crio-62e3f3472d6299a92daed72d5b21433384981d102ea638fce3adb30e114bfcf2 WatchSource:0}: Error finding container 62e3f3472d6299a92daed72d5b21433384981d102ea638fce3adb30e114bfcf2: Status 404 returned error can't find the container with id 62e3f3472d6299a92daed72d5b21433384981d102ea638fce3adb30e114bfcf2
Sep 08 04:17:39.339314 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:39.339273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd" event={"ID":"37987e4d-135f-4185-aee5-2bab62703d66","Type":"ContainerStarted","Data":"dc41cb3e541189cc3e9861a655126338e5d4eec7a33fd4bf7b40fadd1875abcb"}
Sep 08 04:17:39.339692 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:39.339324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd" event={"ID":"37987e4d-135f-4185-aee5-2bab62703d66","Type":"ContainerStarted","Data":"62e3f3472d6299a92daed72d5b21433384981d102ea638fce3adb30e114bfcf2"}
Sep 08 04:17:39.357726 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:39.357675    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd" podStartSLOduration=1.35765839 podStartE2EDuration="1.35765839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:17:39.355591771 +0000 UTC m=+1039.714071322" watchObservedRunningTime="2026-09-08 04:17:39.35765839 +0000 UTC m=+1039.716137949"
Sep 08 04:17:47.218186 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:47.218148    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"]
Sep 08 04:17:47.259417 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:47.259374    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:17:48.462667 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:48.462622    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:17:56.858394 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:17:56.858359    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:18:02.238191 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.238122    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd" podUID="37987e4d-135f-4185-aee5-2bab62703d66" containerName="main" containerID="cri-o://dc41cb3e541189cc3e9861a655126338e5d4eec7a33fd4bf7b40fadd1875abcb" gracePeriod=45
Sep 08 04:18:02.439461 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.439428    2583 generic.go:356] "Generic (PLEG): container finished" podID="37987e4d-135f-4185-aee5-2bab62703d66" containerID="dc41cb3e541189cc3e9861a655126338e5d4eec7a33fd4bf7b40fadd1875abcb" exitCode=0
Sep 08 04:18:02.439609 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.439497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd" event={"ID":"37987e4d-135f-4185-aee5-2bab62703d66","Type":"ContainerDied","Data":"dc41cb3e541189cc3e9861a655126338e5d4eec7a33fd4bf7b40fadd1875abcb"}
Sep 08 04:18:02.497034 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.496984    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:18:02.646405 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646377    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-home\") pod \"37987e4d-135f-4185-aee5-2bab62703d66\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") "
Sep 08 04:18:02.646570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646461    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-model-cache\") pod \"37987e4d-135f-4185-aee5-2bab62703d66\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") "
Sep 08 04:18:02.646570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc8cx\" (UniqueName: \"kubernetes.io/projected/37987e4d-135f-4185-aee5-2bab62703d66-kube-api-access-xc8cx\") pod \"37987e4d-135f-4185-aee5-2bab62703d66\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") "
Sep 08 04:18:02.646570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37987e4d-135f-4185-aee5-2bab62703d66-tls-certs\") pod \"37987e4d-135f-4185-aee5-2bab62703d66\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") "
Sep 08 04:18:02.646570 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-tmp-dir\") pod \"37987e4d-135f-4185-aee5-2bab62703d66\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") "
Sep 08 04:18:02.646789 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646628    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-dshm\") pod \"37987e4d-135f-4185-aee5-2bab62703d66\" (UID: \"37987e4d-135f-4185-aee5-2bab62703d66\") "
Sep 08 04:18:02.646789 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-home" (OuterVolumeSpecName: "home") pod "37987e4d-135f-4185-aee5-2bab62703d66" (UID: "37987e4d-135f-4185-aee5-2bab62703d66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:02.646789 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646764    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-model-cache" (OuterVolumeSpecName: "model-cache") pod "37987e4d-135f-4185-aee5-2bab62703d66" (UID: "37987e4d-135f-4185-aee5-2bab62703d66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:02.646920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646893    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37987e4d-135f-4185-aee5-2bab62703d66" (UID: "37987e4d-135f-4185-aee5-2bab62703d66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:02.646977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646947    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:02.646977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.646960    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:02.648807 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.648785    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37987e4d-135f-4185-aee5-2bab62703d66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37987e4d-135f-4185-aee5-2bab62703d66" (UID: "37987e4d-135f-4185-aee5-2bab62703d66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:18:02.648922 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.648826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37987e4d-135f-4185-aee5-2bab62703d66-kube-api-access-xc8cx" (OuterVolumeSpecName: "kube-api-access-xc8cx") pod "37987e4d-135f-4185-aee5-2bab62703d66" (UID: "37987e4d-135f-4185-aee5-2bab62703d66"). InnerVolumeSpecName "kube-api-access-xc8cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:18:02.648922 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.648866    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-dshm" (OuterVolumeSpecName: "dshm") pod "37987e4d-135f-4185-aee5-2bab62703d66" (UID: "37987e4d-135f-4185-aee5-2bab62703d66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:02.747684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.747614    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc8cx\" (UniqueName: \"kubernetes.io/projected/37987e4d-135f-4185-aee5-2bab62703d66-kube-api-access-xc8cx\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:02.747684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.747642    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37987e4d-135f-4185-aee5-2bab62703d66-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:02.747684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.747673    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:02.747684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:02.747681    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37987e4d-135f-4185-aee5-2bab62703d66-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:03.446384 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:03.446353    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"
Sep 08 04:18:03.446384 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:03.446371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd" event={"ID":"37987e4d-135f-4185-aee5-2bab62703d66","Type":"ContainerDied","Data":"62e3f3472d6299a92daed72d5b21433384981d102ea638fce3adb30e114bfcf2"}
Sep 08 04:18:03.446847 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:03.446419    2583 scope.go:117] "RemoveContainer" containerID="dc41cb3e541189cc3e9861a655126338e5d4eec7a33fd4bf7b40fadd1875abcb"
Sep 08 04:18:03.468146 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:03.468120    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"]
Sep 08 04:18:03.471680 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:03.471659    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-cb3759/explicit-model-cb3759-kserve-786b7bf55f-99txd"]
Sep 08 04:18:04.161169 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:04.161132    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37987e4d-135f-4185-aee5-2bab62703d66" path="/var/lib/kubelet/pods/37987e4d-135f-4185-aee5-2bab62703d66/volumes"
Sep 08 04:18:12.169689 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:12.169654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:18:22.267047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.267016    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"]
Sep 08 04:18:22.267448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.267310    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37987e4d-135f-4185-aee5-2bab62703d66" containerName="main"
Sep 08 04:18:22.267448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.267320    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="37987e4d-135f-4185-aee5-2bab62703d66" containerName="main"
Sep 08 04:18:22.267448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.267381    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="37987e4d-135f-4185-aee5-2bab62703d66" containerName="main"
Sep 08 04:18:22.271179 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.271163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.275252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.275229    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cbda8a\"/\"kube-root-ca.crt\""
Sep 08 04:18:22.275535 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.275520    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cbda8a\"/\"default-dockercfg-bhmlc\""
Sep 08 04:18:22.276235 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.276220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cbda8a\"/\"noten-model-cbda8a-kserve-self-signed-certs\""
Sep 08 04:18:22.276346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.276326    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cbda8a\"/\"openshift-service-ca.crt\""
Sep 08 04:18:22.288010 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.287988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"]
Sep 08 04:18:22.306535 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.306513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tmp-dir\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.306614 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.306564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tls-certs\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.306660 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.306610    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-dshm\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.306660 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.306653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-home\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.306747 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.306669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-model-cache\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.306747 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.306685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbzx\" (UniqueName: \"kubernetes.io/projected/6bb3ff1e-a0ee-4055-be85-62b88f68944b-kube-api-access-6sbzx\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.407572 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.407546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-dshm\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.407669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.407584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-home\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.407669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.407599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-model-cache\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.407669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.407622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbzx\" (UniqueName: \"kubernetes.io/projected/6bb3ff1e-a0ee-4055-be85-62b88f68944b-kube-api-access-6sbzx\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.407669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.407664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tmp-dir\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.407917 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.407733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tls-certs\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.408045 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.408018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-home\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.408105 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.408070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-model-cache\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.408155 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.408137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tmp-dir\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.409948 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.409925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-dshm\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.410186 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.410170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tls-certs\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.418373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.418355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbzx\" (UniqueName: \"kubernetes.io/projected/6bb3ff1e-a0ee-4055-be85-62b88f68944b-kube-api-access-6sbzx\") pod \"noten-model-cbda8a-kserve-696f987b4c-vkrqj\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") " pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.579664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.579611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:22.871870 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:22.871686    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"]
Sep 08 04:18:22.876840 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:18:22.876790    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bb3ff1e_a0ee_4055_be85_62b88f68944b.slice/crio-302aebb1a59fa2ea827721d66a6790bdf210f4729d30c29c63019b3810fe9eee WatchSource:0}: Error finding container 302aebb1a59fa2ea827721d66a6790bdf210f4729d30c29c63019b3810fe9eee: Status 404 returned error can't find the container with id 302aebb1a59fa2ea827721d66a6790bdf210f4729d30c29c63019b3810fe9eee
Sep 08 04:18:23.534493 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:23.534459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj" event={"ID":"6bb3ff1e-a0ee-4055-be85-62b88f68944b","Type":"ContainerStarted","Data":"f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108"}
Sep 08 04:18:23.534863 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:23.534505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj" event={"ID":"6bb3ff1e-a0ee-4055-be85-62b88f68944b","Type":"ContainerStarted","Data":"302aebb1a59fa2ea827721d66a6790bdf210f4729d30c29c63019b3810fe9eee"}
Sep 08 04:18:23.566854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:23.566808    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj" podStartSLOduration=1.566794388 podStartE2EDuration="1.566794388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:18:23.566456442 +0000 UTC m=+1083.924936004" watchObservedRunningTime="2026-09-08 04:18:23.566794388 +0000 UTC m=+1083.925273947"
Sep 08 04:18:25.135151 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:25.135120    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"]
Sep 08 04:18:32.579900 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:32.579867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:37.963333 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:37.963302    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"]
Sep 08 04:18:38.935365 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:38.935327    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"]
Sep 08 04:18:40.561823 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.561724    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj" podUID="6bb3ff1e-a0ee-4055-be85-62b88f68944b" containerName="main" containerID="cri-o://f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108" gracePeriod=45
Sep 08 04:18:40.796238 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.796213    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:40.849363 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849306    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tmp-dir\") pod \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") "
Sep 08 04:18:40.849363 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-dshm\") pod \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") "
Sep 08 04:18:40.849510 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-home\") pod \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") "
Sep 08 04:18:40.849510 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sbzx\" (UniqueName: \"kubernetes.io/projected/6bb3ff1e-a0ee-4055-be85-62b88f68944b-kube-api-access-6sbzx\") pod \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") "
Sep 08 04:18:40.849596 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tls-certs\") pod \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") "
Sep 08 04:18:40.849596 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849548    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-model-cache\") pod \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\" (UID: \"6bb3ff1e-a0ee-4055-be85-62b88f68944b\") "
Sep 08 04:18:40.849596 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849566    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bb3ff1e-a0ee-4055-be85-62b88f68944b" (UID: "6bb3ff1e-a0ee-4055-be85-62b88f68944b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:40.849763 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-home" (OuterVolumeSpecName: "home") pod "6bb3ff1e-a0ee-4055-be85-62b88f68944b" (UID: "6bb3ff1e-a0ee-4055-be85-62b88f68944b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:40.849876 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849855    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:40.849876 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849876    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:40.849980 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.849878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bb3ff1e-a0ee-4055-be85-62b88f68944b" (UID: "6bb3ff1e-a0ee-4055-be85-62b88f68944b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:40.851477 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.851455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-dshm" (OuterVolumeSpecName: "dshm") pod "6bb3ff1e-a0ee-4055-be85-62b88f68944b" (UID: "6bb3ff1e-a0ee-4055-be85-62b88f68944b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:40.852046 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.852019    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bb3ff1e-a0ee-4055-be85-62b88f68944b" (UID: "6bb3ff1e-a0ee-4055-be85-62b88f68944b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:18:40.852117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.852038    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb3ff1e-a0ee-4055-be85-62b88f68944b-kube-api-access-6sbzx" (OuterVolumeSpecName: "kube-api-access-6sbzx") pod "6bb3ff1e-a0ee-4055-be85-62b88f68944b" (UID: "6bb3ff1e-a0ee-4055-be85-62b88f68944b"). InnerVolumeSpecName "kube-api-access-6sbzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:18:40.950518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.950498    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbzx\" (UniqueName: \"kubernetes.io/projected/6bb3ff1e-a0ee-4055-be85-62b88f68944b-kube-api-access-6sbzx\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:40.950518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.950520    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3ff1e-a0ee-4055-be85-62b88f68944b-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:40.950632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.950529    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:40.950632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:40.950538    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3ff1e-a0ee-4055-be85-62b88f68944b-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:41.610088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.610064    2583 generic.go:356] "Generic (PLEG): container finished" podID="6bb3ff1e-a0ee-4055-be85-62b88f68944b" containerID="f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108" exitCode=0
Sep 08 04:18:41.610435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.610099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj" event={"ID":"6bb3ff1e-a0ee-4055-be85-62b88f68944b","Type":"ContainerDied","Data":"f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108"}
Sep 08 04:18:41.610435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.610104    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"
Sep 08 04:18:41.610435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.610125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj" event={"ID":"6bb3ff1e-a0ee-4055-be85-62b88f68944b","Type":"ContainerDied","Data":"302aebb1a59fa2ea827721d66a6790bdf210f4729d30c29c63019b3810fe9eee"}
Sep 08 04:18:41.610435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.610142    2583 scope.go:117] "RemoveContainer" containerID="f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108"
Sep 08 04:18:41.618600 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.618582    2583 scope.go:117] "RemoveContainer" containerID="f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108"
Sep 08 04:18:41.618871 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:18:41.618851    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108\": container with ID starting with f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108 not found: ID does not exist" containerID="f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108"
Sep 08 04:18:41.618954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.618877    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108"} err="failed to get container status \"f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108\": rpc error: code = NotFound desc = could not find container \"f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108\": container with ID starting with f1d14f49847f565bb9e28e1c691bf3f9ea977fbc681c5b6c01447fe038dfa108 not found: ID does not exist"
Sep 08 04:18:41.640725 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.640702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"]
Sep 08 04:18:41.649642 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:41.649618    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-cbda8a/noten-model-cbda8a-kserve-696f987b4c-vkrqj"]
Sep 08 04:18:42.159033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:42.159006    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb3ff1e-a0ee-4055-be85-62b88f68944b" path="/var/lib/kubelet/pods/6bb3ff1e-a0ee-4055-be85-62b88f68944b/volumes"
Sep 08 04:18:42.517705 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:42.517681    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:18:52.982058 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:52.982004    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9" podUID="58270842-164a-40cd-8dce-5413f9dc5e82" containerName="main" containerID="cri-o://f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527" gracePeriod=45
Sep 08 04:18:53.238521 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.238469    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:18:53.339414 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-dshm\") pod \"58270842-164a-40cd-8dce-5413f9dc5e82\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") "
Sep 08 04:18:53.339533 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2v8s\" (UniqueName: \"kubernetes.io/projected/58270842-164a-40cd-8dce-5413f9dc5e82-kube-api-access-g2v8s\") pod \"58270842-164a-40cd-8dce-5413f9dc5e82\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") "
Sep 08 04:18:53.339533 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-tmp-dir\") pod \"58270842-164a-40cd-8dce-5413f9dc5e82\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") "
Sep 08 04:18:53.339533 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-model-cache\") pod \"58270842-164a-40cd-8dce-5413f9dc5e82\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") "
Sep 08 04:18:53.339533 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-home\") pod \"58270842-164a-40cd-8dce-5413f9dc5e82\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") "
Sep 08 04:18:53.339533 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339523    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58270842-164a-40cd-8dce-5413f9dc5e82-tls-certs\") pod \"58270842-164a-40cd-8dce-5413f9dc5e82\" (UID: \"58270842-164a-40cd-8dce-5413f9dc5e82\") "
Sep 08 04:18:53.339831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58270842-164a-40cd-8dce-5413f9dc5e82" (UID: "58270842-164a-40cd-8dce-5413f9dc5e82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:53.339831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-model-cache" (OuterVolumeSpecName: "model-cache") pod "58270842-164a-40cd-8dce-5413f9dc5e82" (UID: "58270842-164a-40cd-8dce-5413f9dc5e82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:53.339831 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.339787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-home" (OuterVolumeSpecName: "home") pod "58270842-164a-40cd-8dce-5413f9dc5e82" (UID: "58270842-164a-40cd-8dce-5413f9dc5e82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:53.341652 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.341631    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58270842-164a-40cd-8dce-5413f9dc5e82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58270842-164a-40cd-8dce-5413f9dc5e82" (UID: "58270842-164a-40cd-8dce-5413f9dc5e82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:18:53.341752 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.341668    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58270842-164a-40cd-8dce-5413f9dc5e82-kube-api-access-g2v8s" (OuterVolumeSpecName: "kube-api-access-g2v8s") pod "58270842-164a-40cd-8dce-5413f9dc5e82" (UID: "58270842-164a-40cd-8dce-5413f9dc5e82"). InnerVolumeSpecName "kube-api-access-g2v8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:18:53.341809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.341732    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-dshm" (OuterVolumeSpecName: "dshm") pod "58270842-164a-40cd-8dce-5413f9dc5e82" (UID: "58270842-164a-40cd-8dce-5413f9dc5e82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:53.440258 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.440233    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:53.440258 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.440256    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58270842-164a-40cd-8dce-5413f9dc5e82-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:53.440392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.440271    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:53.440392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.440282    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2v8s\" (UniqueName: \"kubernetes.io/projected/58270842-164a-40cd-8dce-5413f9dc5e82-kube-api-access-g2v8s\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:53.440392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.440294    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:53.440392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.440306    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58270842-164a-40cd-8dce-5413f9dc5e82-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:53.666211 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.666184    2583 generic.go:356] "Generic (PLEG): container finished" podID="58270842-164a-40cd-8dce-5413f9dc5e82" containerID="f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527" exitCode=0
Sep 08 04:18:53.666320 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.666218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9" event={"ID":"58270842-164a-40cd-8dce-5413f9dc5e82","Type":"ContainerDied","Data":"f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527"}
Sep 08 04:18:53.666320 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.666227    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"
Sep 08 04:18:53.666320 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.666253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9" event={"ID":"58270842-164a-40cd-8dce-5413f9dc5e82","Type":"ContainerDied","Data":"6ddd95206f2f5244f25b12d0c53d6aacb4619f5300a91589d5889724f7f88256"}
Sep 08 04:18:53.666320 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.666277    2583 scope.go:117] "RemoveContainer" containerID="f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527"
Sep 08 04:18:53.674960 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.674943    2583 scope.go:117] "RemoveContainer" containerID="f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527"
Sep 08 04:18:53.675200 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:18:53.675177    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527\": container with ID starting with f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527 not found: ID does not exist" containerID="f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527"
Sep 08 04:18:53.675286 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.675204    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527"} err="failed to get container status \"f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527\": rpc error: code = NotFound desc = could not find container \"f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527\": container with ID starting with f4b098fecc10faeab0e9d1809fcc1affb960a2539c339abf9ce34ac360561527 not found: ID does not exist"
Sep 08 04:18:53.707122 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.707103    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"]
Sep 08 04:18:53.710943 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.710924    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ff0f3e/test-model-ff0f3e-kserve-5f7bf7bfc-kkcv9"]
Sep 08 04:18:53.954399 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:53.954316    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm" podUID="e8c32540-f7c1-4926-b58f-d57b1dd7fc50" containerName="main" containerID="cri-o://31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721" gracePeriod=45
Sep 08 04:18:54.159926 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.159894    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58270842-164a-40cd-8dce-5413f9dc5e82" path="/var/lib/kubelet/pods/58270842-164a-40cd-8dce-5413f9dc5e82/volumes"
Sep 08 04:18:54.187950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.187930    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:18:54.247318 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-model-cache\") pod \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") "
Sep 08 04:18:54.247318 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247299    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tmp-dir\") pod \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") "
Sep 08 04:18:54.247484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k68m7\" (UniqueName: \"kubernetes.io/projected/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-kube-api-access-k68m7\") pod \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") "
Sep 08 04:18:54.247484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247357    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-home\") pod \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") "
Sep 08 04:18:54.247484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tls-certs\") pod \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") "
Sep 08 04:18:54.247484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247432    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-dshm\") pod \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\" (UID: \"e8c32540-f7c1-4926-b58f-d57b1dd7fc50\") "
Sep 08 04:18:54.247774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8c32540-f7c1-4926-b58f-d57b1dd7fc50" (UID: "e8c32540-f7c1-4926-b58f-d57b1dd7fc50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:54.247774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247578    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8c32540-f7c1-4926-b58f-d57b1dd7fc50" (UID: "e8c32540-f7c1-4926-b58f-d57b1dd7fc50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:54.247774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247651    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-home" (OuterVolumeSpecName: "home") pod "e8c32540-f7c1-4926-b58f-d57b1dd7fc50" (UID: "e8c32540-f7c1-4926-b58f-d57b1dd7fc50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:54.247774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247680    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:54.247774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.247702    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:54.249509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.249485    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8c32540-f7c1-4926-b58f-d57b1dd7fc50" (UID: "e8c32540-f7c1-4926-b58f-d57b1dd7fc50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:18:54.249509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.249500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-dshm" (OuterVolumeSpecName: "dshm") pod "e8c32540-f7c1-4926-b58f-d57b1dd7fc50" (UID: "e8c32540-f7c1-4926-b58f-d57b1dd7fc50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:18:54.249614 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.249513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-kube-api-access-k68m7" (OuterVolumeSpecName: "kube-api-access-k68m7") pod "e8c32540-f7c1-4926-b58f-d57b1dd7fc50" (UID: "e8c32540-f7c1-4926-b58f-d57b1dd7fc50"). InnerVolumeSpecName "kube-api-access-k68m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:18:54.348676 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.348655    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k68m7\" (UniqueName: \"kubernetes.io/projected/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-kube-api-access-k68m7\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:54.348676 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.348675    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:54.348830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.348685    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:54.348830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.348694    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c32540-f7c1-4926-b58f-d57b1dd7fc50-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:18:54.672971 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.672945    2583 generic.go:356] "Generic (PLEG): container finished" podID="e8c32540-f7c1-4926-b58f-d57b1dd7fc50" containerID="31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721" exitCode=0
Sep 08 04:18:54.673096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.672988    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"
Sep 08 04:18:54.673096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.673031    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm" event={"ID":"e8c32540-f7c1-4926-b58f-d57b1dd7fc50","Type":"ContainerDied","Data":"31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721"}
Sep 08 04:18:54.673096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.673073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm" event={"ID":"e8c32540-f7c1-4926-b58f-d57b1dd7fc50","Type":"ContainerDied","Data":"da17f4250ae86f5dd906047cef452a449f09d003f070806e7b9d4749fd68c673"}
Sep 08 04:18:54.673096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.673093    2583 scope.go:117] "RemoveContainer" containerID="31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721"
Sep 08 04:18:54.681394 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.681378    2583 scope.go:117] "RemoveContainer" containerID="31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721"
Sep 08 04:18:54.681651 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:18:54.681636    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721\": container with ID starting with 31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721 not found: ID does not exist" containerID="31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721"
Sep 08 04:18:54.681692 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.681656    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721"} err="failed to get container status \"31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721\": rpc error: code = NotFound desc = could not find container \"31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721\": container with ID starting with 31eec996823108c967c70c13e380e5fb5efd5dcb52f7ae9a99f527bcd9239721 not found: ID does not exist"
Sep 08 04:18:54.714835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.714812    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"]
Sep 08 04:18:54.724269 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:54.724245    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cf5ae2/test-model-cf5ae2-kserve-9db94c749-kjtgm"]
Sep 08 04:18:55.388703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:55.388664    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:18:56.158835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:18:56.158800    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c32540-f7c1-4926-b58f-d57b1dd7fc50" path="/var/lib/kubelet/pods/e8c32540-f7c1-4926-b58f-d57b1dd7fc50/volumes"
Sep 08 04:19:11.846266 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:11.846230    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"]
Sep 08 04:19:11.846658 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:11.846405    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" podUID="c9935240-fd0e-4685-818e-b1dd07a5913e" containerName="maas-api" containerID="cri-o://67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c" gracePeriod=30
Sep 08 04:19:11.917306 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:11.917280    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"]
Sep 08 04:19:11.934608 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:11.934585    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cf5ae2-29814015-nw6zn"]
Sep 08 04:19:11.974818 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:11.974794    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"]
Sep 08 04:19:11.975039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:11.975012    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw" podUID="3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" containerName="payload-processing" containerID="cri-o://323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835" gracePeriod=30
Sep 08 04:19:12.023101 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.023072    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"]
Sep 08 04:19:12.023340 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.023297    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h" podUID="d4c66763-c9ed-45fb-b024-07e242ebba69" containerName="payload-pre-processing" containerID="cri-o://ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4" gracePeriod=30
Sep 08 04:19:12.144857 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.144833    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:19:12.161403 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.161361    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caccbac6-9139-4e2f-bc8e-79ba3160c7c1" path="/var/lib/kubelet/pods/caccbac6-9139-4e2f-bc8e-79ba3160c7c1/volumes"
Sep 08 04:19:12.274502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.274480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:19:12.280074 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.280056    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:19:12.284247 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.284231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-metrics-certs\") pod \"c9935240-fd0e-4685-818e-b1dd07a5913e\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") "
Sep 08 04:19:12.284327 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.284316    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls\") pod \"c9935240-fd0e-4685-818e-b1dd07a5913e\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") "
Sep 08 04:19:12.284363 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.284347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9935240-fd0e-4685-818e-b1dd07a5913e-openshift-service-ca\") pod \"c9935240-fd0e-4685-818e-b1dd07a5913e\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") "
Sep 08 04:19:12.284405 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.284364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk6md\" (UniqueName: \"kubernetes.io/projected/c9935240-fd0e-4685-818e-b1dd07a5913e-kube-api-access-xk6md\") pod \"c9935240-fd0e-4685-818e-b1dd07a5913e\" (UID: \"c9935240-fd0e-4685-818e-b1dd07a5913e\") "
Sep 08 04:19:12.284720 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.284685    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9935240-fd0e-4685-818e-b1dd07a5913e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9935240-fd0e-4685-818e-b1dd07a5913e" (UID: "c9935240-fd0e-4685-818e-b1dd07a5913e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:19:12.286308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.286284    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9935240-fd0e-4685-818e-b1dd07a5913e-kube-api-access-xk6md" (OuterVolumeSpecName: "kube-api-access-xk6md") pod "c9935240-fd0e-4685-818e-b1dd07a5913e" (UID: "c9935240-fd0e-4685-818e-b1dd07a5913e"). InnerVolumeSpecName "kube-api-access-xk6md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:19:12.286400 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.286367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9935240-fd0e-4685-818e-b1dd07a5913e" (UID: "c9935240-fd0e-4685-818e-b1dd07a5913e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:19:12.286448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.286425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9935240-fd0e-4685-818e-b1dd07a5913e" (UID: "c9935240-fd0e-4685-818e-b1dd07a5913e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:19:12.385557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-plugins-config-volume\") pod \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") "
Sep 08 04:19:12.385557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc2nd\" (UniqueName: \"kubernetes.io/projected/d4c66763-c9ed-45fb-b024-07e242ebba69-kube-api-access-mc2nd\") pod \"d4c66763-c9ed-45fb-b024-07e242ebba69\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") "
Sep 08 04:19:12.385835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bbwx\" (UniqueName: \"kubernetes.io/projected/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-kube-api-access-6bbwx\") pod \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\" (UID: \"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5\") "
Sep 08 04:19:12.385835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385605    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c66763-c9ed-45fb-b024-07e242ebba69-plugins-config-volume\") pod \"d4c66763-c9ed-45fb-b024-07e242ebba69\" (UID: \"d4c66763-c9ed-45fb-b024-07e242ebba69\") "
Sep 08 04:19:12.385950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385854    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9935240-fd0e-4685-818e-b1dd07a5913e-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.385950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385900    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6md\" (UniqueName: \"kubernetes.io/projected/c9935240-fd0e-4685-818e-b1dd07a5913e-kube-api-access-xk6md\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.385950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385918    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.385950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385924    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" (UID: "3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:19:12.386157 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.385933    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9935240-fd0e-4685-818e-b1dd07a5913e-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.386157 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.386136    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c66763-c9ed-45fb-b024-07e242ebba69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4c66763-c9ed-45fb-b024-07e242ebba69" (UID: "d4c66763-c9ed-45fb-b024-07e242ebba69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:19:12.388028 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.387999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-kube-api-access-6bbwx" (OuterVolumeSpecName: "kube-api-access-6bbwx") pod "3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" (UID: "3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5"). InnerVolumeSpecName "kube-api-access-6bbwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:19:12.388235 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.388210    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c66763-c9ed-45fb-b024-07e242ebba69-kube-api-access-mc2nd" (OuterVolumeSpecName: "kube-api-access-mc2nd") pod "d4c66763-c9ed-45fb-b024-07e242ebba69" (UID: "d4c66763-c9ed-45fb-b024-07e242ebba69"). InnerVolumeSpecName "kube-api-access-mc2nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:19:12.486609 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.486586    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.486609 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.486611    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc2nd\" (UniqueName: \"kubernetes.io/projected/d4c66763-c9ed-45fb-b024-07e242ebba69-kube-api-access-mc2nd\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.486609 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.486624    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbwx\" (UniqueName: \"kubernetes.io/projected/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5-kube-api-access-6bbwx\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.486804 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.486637    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c66763-c9ed-45fb-b024-07e242ebba69-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:19:12.753865 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.753833    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4c66763-c9ed-45fb-b024-07e242ebba69" containerID="ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4" exitCode=0
Sep 08 04:19:12.753997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.753883    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"
Sep 08 04:19:12.753997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.753920    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h" event={"ID":"d4c66763-c9ed-45fb-b024-07e242ebba69","Type":"ContainerDied","Data":"ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4"}
Sep 08 04:19:12.753997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.753980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h" event={"ID":"d4c66763-c9ed-45fb-b024-07e242ebba69","Type":"ContainerDied","Data":"f3e8bff856864be2126c0f54294bd63716a1f195f897199b4c8d4578161b9ab9"}
Sep 08 04:19:12.754143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.754009    2583 scope.go:117] "RemoveContainer" containerID="ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4"
Sep 08 04:19:12.755200 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.755183    2583 generic.go:356] "Generic (PLEG): container finished" podID="c9935240-fd0e-4685-818e-b1dd07a5913e" containerID="67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c" exitCode=0
Sep 08 04:19:12.755270 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.755236    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"
Sep 08 04:19:12.755315 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.755276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" event={"ID":"c9935240-fd0e-4685-818e-b1dd07a5913e","Type":"ContainerDied","Data":"67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c"}
Sep 08 04:19:12.755364 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.755331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86" event={"ID":"c9935240-fd0e-4685-818e-b1dd07a5913e","Type":"ContainerDied","Data":"2f3a19477b08beeec72fc5cfa3ef618d5afe9269f7f918591c805930e0373409"}
Sep 08 04:19:12.756655 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.756631    2583 generic.go:356] "Generic (PLEG): container finished" podID="3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" containerID="323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835" exitCode=0
Sep 08 04:19:12.756798 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.756666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw" event={"ID":"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5","Type":"ContainerDied","Data":"323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835"}
Sep 08 04:19:12.756798 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.756677    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"
Sep 08 04:19:12.756798 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.756703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw" event={"ID":"3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5","Type":"ContainerDied","Data":"4f0b77e90cf2a67e557de10d4fb7cce6f96539f9912a0976f6d2484731326c90"}
Sep 08 04:19:12.763169 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.763151    2583 scope.go:117] "RemoveContainer" containerID="ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4"
Sep 08 04:19:12.763452 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:19:12.763431    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4\": container with ID starting with ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4 not found: ID does not exist" containerID="ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4"
Sep 08 04:19:12.763541 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.763458    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4"} err="failed to get container status \"ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4\": rpc error: code = NotFound desc = could not find container \"ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4\": container with ID starting with ea88fbb46e9d4d5501549b2ca7038d50bd0af66dfd3fa73e7dead223609a3cc4 not found: ID does not exist"
Sep 08 04:19:12.763541 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.763474    2583 scope.go:117] "RemoveContainer" containerID="67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c"
Sep 08 04:19:12.772720 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.772705    2583 scope.go:117] "RemoveContainer" containerID="67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c"
Sep 08 04:19:12.773055 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:19:12.773036    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c\": container with ID starting with 67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c not found: ID does not exist" containerID="67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c"
Sep 08 04:19:12.773105 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.773062    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c"} err="failed to get container status \"67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c\": rpc error: code = NotFound desc = could not find container \"67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c\": container with ID starting with 67fa059a8e1c7c6ac1bb97236c4473720676c907ec25bc17616888734b074d7c not found: ID does not exist"
Sep 08 04:19:12.773105 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.773077    2583 scope.go:117] "RemoveContainer" containerID="323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835"
Sep 08 04:19:12.780088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.780073    2583 scope.go:117] "RemoveContainer" containerID="323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835"
Sep 08 04:19:12.780321 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:19:12.780307    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835\": container with ID starting with 323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835 not found: ID does not exist" containerID="323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835"
Sep 08 04:19:12.780364 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.780327    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835"} err="failed to get container status \"323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835\": rpc error: code = NotFound desc = could not find container \"323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835\": container with ID starting with 323f358e4e7b6ccf1e8150d98bc181e4d7db2686d78e2505ab59f0e3026a8835 not found: ID does not exist"
Sep 08 04:19:12.824914 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.824888    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"]
Sep 08 04:19:12.828726 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.828706    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf5ae2-cb4b5f54c-bxhlw"]
Sep 08 04:19:12.854802 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.854781    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"]
Sep 08 04:19:12.864888 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.864868    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf5ae2-575494c466-bn24h"]
Sep 08 04:19:12.881788 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.881766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"]
Sep 08 04:19:12.899264 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:12.899241    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf5ae2-68f5cb6d79-6ff86"]
Sep 08 04:19:14.159472 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:14.159439    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" path="/var/lib/kubelet/pods/3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5/volumes"
Sep 08 04:19:14.159848 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:14.159799    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9935240-fd0e-4685-818e-b1dd07a5913e" path="/var/lib/kubelet/pods/c9935240-fd0e-4685-818e-b1dd07a5913e/volumes"
Sep 08 04:19:14.160101 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:14.160088    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4c66763-c9ed-45fb-b024-07e242ebba69" path="/var/lib/kubelet/pods/d4c66763-c9ed-45fb-b024-07e242ebba69/volumes"
Sep 08 04:19:15.137888 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:15.137854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:19:19.958596 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"]
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958867    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" containerName="payload-processing"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958880    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" containerName="payload-processing"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958892    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9935240-fd0e-4685-818e-b1dd07a5913e" containerName="maas-api"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958901    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9935240-fd0e-4685-818e-b1dd07a5913e" containerName="maas-api"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb3ff1e-a0ee-4055-be85-62b88f68944b" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958917    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb3ff1e-a0ee-4055-be85-62b88f68944b" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958928    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c66763-c9ed-45fb-b024-07e242ebba69" containerName="payload-pre-processing"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958933    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c66763-c9ed-45fb-b024-07e242ebba69" containerName="payload-pre-processing"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958940    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58270842-164a-40cd-8dce-5413f9dc5e82" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958945    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="58270842-164a-40cd-8dce-5413f9dc5e82" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958954    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c32540-f7c1-4926-b58f-d57b1dd7fc50" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.958959    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c32540-f7c1-4926-b58f-d57b1dd7fc50" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.959014    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3469a1f2-461a-4f49-ba5b-d9d21dc6ccf5" containerName="payload-processing"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.959022    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb3ff1e-a0ee-4055-be85-62b88f68944b" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.959028    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c32540-f7c1-4926-b58f-d57b1dd7fc50" containerName="main"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.959034    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c66763-c9ed-45fb-b024-07e242ebba69" containerName="payload-pre-processing"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.959040    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9935240-fd0e-4685-818e-b1dd07a5913e" containerName="maas-api"
Sep 08 04:19:19.959039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.959046    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="58270842-164a-40cd-8dce-5413f9dc5e82" containerName="main"
Sep 08 04:19:19.961722 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.961703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:19.967324 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.967298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-75264d-serving-cert\""
Sep 08 04:19:19.971068 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.971048    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 04:19:19.981031 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.981006    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npqmj\""
Sep 08 04:19:19.982416 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.982395    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 04:19:19.983636 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.983615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"]
Sep 08 04:19:19.985414 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:19.985393    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:19:20.037032 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.037010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.037116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.037063    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.037116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.037090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bb77\" (UniqueName: \"kubernetes.io/projected/453a20f6-c45e-4ff9-ab98-7eb1add12062-kube-api-access-8bb77\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.037116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.037112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453a20f6-c45e-4ff9-ab98-7eb1add12062-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.138129 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.138109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.138239 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.138150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.138239 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.138167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bb77\" (UniqueName: \"kubernetes.io/projected/453a20f6-c45e-4ff9-ab98-7eb1add12062-kube-api-access-8bb77\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.138363 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.138283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453a20f6-c45e-4ff9-ab98-7eb1add12062-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.143040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.143022    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-75264d-serving-cert\""
Sep 08 04:19:20.143091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.143040    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:19:20.143141 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.143126    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 04:19:20.148702 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:19:20.148688    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-serving-cert: secret "maas-api-e2e-shared-a-w2-75264d-serving-cert" not found
Sep 08 04:19:20.148787 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:19:20.148775    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls podName:453a20f6-c45e-4ff9-ab98-7eb1add12062 nodeName:}" failed. No retries permitted until 2026-09-08 04:19:20.648733059 +0000 UTC m=+1141.007212597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls") pod "maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" (UID: "453a20f6-c45e-4ff9-ab98-7eb1add12062") : secret "maas-api-e2e-shared-a-w2-75264d-serving-cert" not found
Sep 08 04:19:20.149167 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.149148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453a20f6-c45e-4ff9-ab98-7eb1add12062-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.150835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.150817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.171224 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.171202    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 04:19:20.174394 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.174371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bb77\" (UniqueName: \"kubernetes.io/projected/453a20f6-c45e-4ff9-ab98-7eb1add12062-kube-api-access-8bb77\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.743009 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.742983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.745242 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.745217    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:20.874287 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.874265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npqmj\""
Sep 08 04:19:20.882021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:20.882004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:21.188048 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:21.188018    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"]
Sep 08 04:19:21.199630 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:19:21.199602    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453a20f6_c45e_4ff9_ab98_7eb1add12062.slice/crio-e18a2edba730d2da8417180dc43a662601a83a0c04775289b2419d677bcc0724 WatchSource:0}: Error finding container e18a2edba730d2da8417180dc43a662601a83a0c04775289b2419d677bcc0724: Status 404 returned error can't find the container with id e18a2edba730d2da8417180dc43a662601a83a0c04775289b2419d677bcc0724
Sep 08 04:19:21.202783 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:21.201310    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 04:19:21.797554 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:21.797522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" event={"ID":"453a20f6-c45e-4ff9-ab98-7eb1add12062","Type":"ContainerStarted","Data":"3757f74f36c8252051a2b7791dc3878e96e2a87290633c294052f29763a542b7"}
Sep 08 04:19:21.797679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:21.797608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" event={"ID":"453a20f6-c45e-4ff9-ab98-7eb1add12062","Type":"ContainerStarted","Data":"e18a2edba730d2da8417180dc43a662601a83a0c04775289b2419d677bcc0724"}
Sep 08 04:19:21.797679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:21.797637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:21.847370 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:21.847301    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" podStartSLOduration=2.4245613 podStartE2EDuration="2.84728643s" totalImagesPullingTime="422.72513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:21.846488879 +0000 UTC m=+1142.204968440" watchObservedRunningTime="2026-09-08 04:19:21.84728643 +0000 UTC m=+1142.205765986"
Sep 08 04:19:24.698511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:24.698471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:19:27.806407 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:27.806375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:19:30.169442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.169410    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"]
Sep 08 04:19:30.172673 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.172657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.176993 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.176971    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c2a8bb-dockercfg-q7s9l\""
Sep 08 04:19:30.177115 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.177040    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c2a8bb\""
Sep 08 04:19:30.195336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.195313    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"]
Sep 08 04:19:30.315528 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.315486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ede1a61-2715-4569-9193-ef8b674dd085-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.315638 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.315535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwg22\" (UniqueName: \"kubernetes.io/projected/6ede1a61-2715-4569-9193-ef8b674dd085-kube-api-access-rwg22\") pod \"payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.415946 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.415918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ede1a61-2715-4569-9193-ef8b674dd085-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.416055 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.415957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwg22\" (UniqueName: \"kubernetes.io/projected/6ede1a61-2715-4569-9193-ef8b674dd085-kube-api-access-rwg22\") pod \"payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.416626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.416609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ede1a61-2715-4569-9193-ef8b674dd085-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.431228 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.431180    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwg22\" (UniqueName: \"kubernetes.io/projected/6ede1a61-2715-4569-9193-ef8b674dd085-kube-api-access-rwg22\") pod \"payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.483190 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.483152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.764173 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.764148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"]
Sep 08 04:19:30.766676 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:19:30.766642    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ede1a61_2715_4569_9193_ef8b674dd085.slice/crio-979db62c54f0485b7490293415ede579342fdd014807d60367e741c38db35685 WatchSource:0}: Error finding container 979db62c54f0485b7490293415ede579342fdd014807d60367e741c38db35685: Status 404 returned error can't find the container with id 979db62c54f0485b7490293415ede579342fdd014807d60367e741c38db35685
Sep 08 04:19:30.836378 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.836350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87" event={"ID":"6ede1a61-2715-4569-9193-ef8b674dd085","Type":"ContainerStarted","Data":"a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3"}
Sep 08 04:19:30.836502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.836406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87" event={"ID":"6ede1a61-2715-4569-9193-ef8b674dd085","Type":"ContainerStarted","Data":"979db62c54f0485b7490293415ede579342fdd014807d60367e741c38db35685"}
Sep 08 04:19:30.836502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.836436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:30.868955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:30.868910    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87" podStartSLOduration=0.868894728 podStartE2EDuration="868.894728ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:30.867342402 +0000 UTC m=+1151.225821961" watchObservedRunningTime="2026-09-08 04:19:30.868894728 +0000 UTC m=+1151.227374288"
Sep 08 04:19:37.466653 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:37.466621    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:19:41.842229 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:41.842200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:19:48.228720 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.228640    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"]
Sep 08 04:19:48.232437 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.232415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.235055 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.235031    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c2eafd-serving-cert\""
Sep 08 04:19:48.245296 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.245270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txpbz\" (UniqueName: \"kubernetes.io/projected/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-kube-api-access-txpbz\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.245412 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.245309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.245412 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.245333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.245535 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.245410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.253029 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.253006    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"]
Sep 08 04:19:48.346730 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.346700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.346913 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.346813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txpbz\" (UniqueName: \"kubernetes.io/projected/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-kube-api-access-txpbz\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.346913 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.346853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.347040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.346911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.347478 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.347452    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.348975 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.348955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.349156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.349138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.363454 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.363429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txpbz\" (UniqueName: \"kubernetes.io/projected/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-kube-api-access-txpbz\") pod \"maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.543216 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.543166    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:48.818395 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.818371    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"]
Sep 08 04:19:48.820236 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:19:48.820211    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279ffc93_94f5_4e85_b0f3_2e4db4b7bb6d.slice/crio-27e373d48eb22c5eba0bed1ab36ec4740648e449006ac3cc57588348c16e0351 WatchSource:0}: Error finding container 27e373d48eb22c5eba0bed1ab36ec4740648e449006ac3cc57588348c16e0351: Status 404 returned error can't find the container with id 27e373d48eb22c5eba0bed1ab36ec4740648e449006ac3cc57588348c16e0351
Sep 08 04:19:48.914778 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:48.914727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69" event={"ID":"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d","Type":"ContainerStarted","Data":"27e373d48eb22c5eba0bed1ab36ec4740648e449006ac3cc57588348c16e0351"}
Sep 08 04:19:49.121262 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.121179    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"]
Sep 08 04:19:49.124268 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.124252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.130787 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.130757    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c2eafd-dockercfg-gwpns\""
Sep 08 04:19:49.132524 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.132509    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c2eafd\""
Sep 08 04:19:49.153892 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.153870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288b1513-588b-4753-aef3-09dee16321dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.153984 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.153929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxwq\" (UniqueName: \"kubernetes.io/projected/288b1513-588b-4753-aef3-09dee16321dc-kube-api-access-vjxwq\") pod \"payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.158705 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.158683    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"]
Sep 08 04:19:49.217783 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.217761    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"]
Sep 08 04:19:49.220973 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.220957    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.243333 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.243310    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"]
Sep 08 04:19:49.254490 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.254294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288b1513-588b-4753-aef3-09dee16321dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.254490 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.254340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.254632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.254546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxwq\" (UniqueName: \"kubernetes.io/projected/288b1513-588b-4753-aef3-09dee16321dc-kube-api-access-vjxwq\") pod \"payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.254632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.254580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9km\" (UniqueName: \"kubernetes.io/projected/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-kube-api-access-xg9km\") pod \"payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.255329 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.255277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288b1513-588b-4753-aef3-09dee16321dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.276168 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.276148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxwq\" (UniqueName: \"kubernetes.io/projected/288b1513-588b-4753-aef3-09dee16321dc-kube-api-access-vjxwq\") pod \"payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.355716 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.355687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.355959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.355791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9km\" (UniqueName: \"kubernetes.io/projected/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-kube-api-access-xg9km\") pod \"payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.356598 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.356578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.370120 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.370099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9km\" (UniqueName: \"kubernetes.io/projected/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-kube-api-access-xg9km\") pod \"payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.434957 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.434897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.530037 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.530007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.747466 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.747200    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"]
Sep 08 04:19:49.749805 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:19:49.749778    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288b1513_588b_4753_aef3_09dee16321dc.slice/crio-22951c6c05835ee4ed1a1e144f04c87e7849287441d734526d58866991176486 WatchSource:0}: Error finding container 22951c6c05835ee4ed1a1e144f04c87e7849287441d734526d58866991176486: Status 404 returned error can't find the container with id 22951c6c05835ee4ed1a1e144f04c87e7849287441d734526d58866991176486
Sep 08 04:19:49.828100 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.828078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"]
Sep 08 04:19:49.830434 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:19:49.830410    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28057ce3_03cf_4f45_91a0_a9c4d6c1d68a.slice/crio-bdfd4ea4e5fbe595fdbee3c31202e456f90768d9d3f6e17291f001303be2bbe6 WatchSource:0}: Error finding container bdfd4ea4e5fbe595fdbee3c31202e456f90768d9d3f6e17291f001303be2bbe6: Status 404 returned error can't find the container with id bdfd4ea4e5fbe595fdbee3c31202e456f90768d9d3f6e17291f001303be2bbe6
Sep 08 04:19:49.926145 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.926110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d" event={"ID":"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a","Type":"ContainerStarted","Data":"46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe"}
Sep 08 04:19:49.926327 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.926177    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:19:49.926327 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.926201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d" event={"ID":"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a","Type":"ContainerStarted","Data":"bdfd4ea4e5fbe595fdbee3c31202e456f90768d9d3f6e17291f001303be2bbe6"}
Sep 08 04:19:49.927851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.927785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69" event={"ID":"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d","Type":"ContainerStarted","Data":"1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f"}
Sep 08 04:19:49.927851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.927847    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:19:49.929379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.929354    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m" event={"ID":"288b1513-588b-4753-aef3-09dee16321dc","Type":"ContainerStarted","Data":"37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51"}
Sep 08 04:19:49.929490 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.929399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m" event={"ID":"288b1513-588b-4753-aef3-09dee16321dc","Type":"ContainerStarted","Data":"22951c6c05835ee4ed1a1e144f04c87e7849287441d734526d58866991176486"}
Sep 08 04:19:49.929490 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.929477    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:19:49.960285 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.960237    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d" podStartSLOduration=0.96022312 podStartE2EDuration="960.22312ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:49.958927676 +0000 UTC m=+1170.317407236" watchObservedRunningTime="2026-09-08 04:19:49.96022312 +0000 UTC m=+1170.318702680"
Sep 08 04:19:49.990360 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:49.990312    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m" podStartSLOduration=0.990296438 podStartE2EDuration="990.296438ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:49.988933001 +0000 UTC m=+1170.347412561" watchObservedRunningTime="2026-09-08 04:19:49.990296438 +0000 UTC m=+1170.348775997"
Sep 08 04:19:50.014797 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:50.014751    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69" podStartSLOduration=1.612221995 podStartE2EDuration="2.014723228s" totalImagesPullingTime="402.501233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:50.013579729 +0000 UTC m=+1170.372059292" watchObservedRunningTime="2026-09-08 04:19:50.014723228 +0000 UTC m=+1170.373202787"
Sep 08 04:19:52.631792 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:52.631752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:19:55.940520 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:19:55.940490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:20:00.936958 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:00.936920    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:20:00.937379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:00.937095    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:20:09.861461 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:09.861423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:20:12.728761 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.728713    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:12.733164 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.733146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:12.735835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.735809    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nwvtr\""
Sep 08 04:20:12.744122 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.744088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:12.854106 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.854074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56vh\" (UniqueName: \"kubernetes.io/projected/9e370acc-ac32-4b41-a539-ba7e99062275-kube-api-access-r56vh\") pod \"test-curl-7840-140093167540384\" (UID: \"9e370acc-ac32-4b41-a539-ba7e99062275\") " pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:12.955244 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.955214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56vh\" (UniqueName: \"kubernetes.io/projected/9e370acc-ac32-4b41-a539-ba7e99062275-kube-api-access-r56vh\") pod \"test-curl-7840-140093167540384\" (UID: \"9e370acc-ac32-4b41-a539-ba7e99062275\") " pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:12.965577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:12.965552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56vh\" (UniqueName: \"kubernetes.io/projected/9e370acc-ac32-4b41-a539-ba7e99062275-kube-api-access-r56vh\") pod \"test-curl-7840-140093167540384\" (UID: \"9e370acc-ac32-4b41-a539-ba7e99062275\") " pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:13.042017 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:13.041994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:13.310645 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:13.310619    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:13.312879 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:20:13.312848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e370acc_ac32_4b41_a539_ba7e99062275.slice/crio-0e94fe899e2d6719bf034fcdaee4335425a1352393840ead5e1d84b4be2f6b96 WatchSource:0}: Error finding container 0e94fe899e2d6719bf034fcdaee4335425a1352393840ead5e1d84b4be2f6b96: Status 404 returned error can't find the container with id 0e94fe899e2d6719bf034fcdaee4335425a1352393840ead5e1d84b4be2f6b96
Sep 08 04:20:14.043112 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:14.043075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540384" event={"ID":"9e370acc-ac32-4b41-a539-ba7e99062275","Type":"ContainerStarted","Data":"0e94fe899e2d6719bf034fcdaee4335425a1352393840ead5e1d84b4be2f6b96"}
Sep 08 04:20:15.049589 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:15.049556    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e370acc-ac32-4b41-a539-ba7e99062275" containerID="490a2d3b1b0bb50c9845abc2c9db9804276adafba614411db499cfb47f5be262" exitCode=0
Sep 08 04:20:15.050024 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:15.049625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540384" event={"ID":"9e370acc-ac32-4b41-a539-ba7e99062275","Type":"ContainerDied","Data":"490a2d3b1b0bb50c9845abc2c9db9804276adafba614411db499cfb47f5be262"}
Sep 08 04:20:16.183683 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.183656    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:16.252339 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.252312    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7840-140093167540384_9e370acc-ac32-4b41-a539-ba7e99062275/test-curl-7840-140093167540384/0.log"
Sep 08 04:20:16.281301 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.281273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r56vh\" (UniqueName: \"kubernetes.io/projected/9e370acc-ac32-4b41-a539-ba7e99062275-kube-api-access-r56vh\") pod \"9e370acc-ac32-4b41-a539-ba7e99062275\" (UID: \"9e370acc-ac32-4b41-a539-ba7e99062275\") "
Sep 08 04:20:16.281764 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.281730    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:16.283201 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.283176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e370acc-ac32-4b41-a539-ba7e99062275-kube-api-access-r56vh" (OuterVolumeSpecName: "kube-api-access-r56vh") pod "9e370acc-ac32-4b41-a539-ba7e99062275" (UID: "9e370acc-ac32-4b41-a539-ba7e99062275"). InnerVolumeSpecName "kube-api-access-r56vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:20:16.289280 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.289259    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:16.382143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.382059    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56vh\" (UniqueName: \"kubernetes.io/projected/9e370acc-ac32-4b41-a539-ba7e99062275-kube-api-access-r56vh\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:20:16.419109 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.419074    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:16.419460 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.419441    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e370acc-ac32-4b41-a539-ba7e99062275" containerName="test-curl-7840-140093167540384"
Sep 08 04:20:16.419460 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.419459    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e370acc-ac32-4b41-a539-ba7e99062275" containerName="test-curl-7840-140093167540384"
Sep 08 04:20:16.419595 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.419527    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e370acc-ac32-4b41-a539-ba7e99062275" containerName="test-curl-7840-140093167540384"
Sep 08 04:20:16.422286 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.422270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:16.438481 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.438454    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:16.483242 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.483218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzkg\" (UniqueName: \"kubernetes.io/projected/cce1be48-6584-418d-a882-c30ad29e4f22-kube-api-access-6gzkg\") pod \"test-curl-7840-140093167540384\" (UID: \"cce1be48-6584-418d-a882-c30ad29e4f22\") " pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:16.584402 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.584377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzkg\" (UniqueName: \"kubernetes.io/projected/cce1be48-6584-418d-a882-c30ad29e4f22-kube-api-access-6gzkg\") pod \"test-curl-7840-140093167540384\" (UID: \"cce1be48-6584-418d-a882-c30ad29e4f22\") " pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:16.596350 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.596329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzkg\" (UniqueName: \"kubernetes.io/projected/cce1be48-6584-418d-a882-c30ad29e4f22-kube-api-access-6gzkg\") pod \"test-curl-7840-140093167540384\" (UID: \"cce1be48-6584-418d-a882-c30ad29e4f22\") " pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:16.731247 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:16.731188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:17.020994 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:17.020962    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:17.023612 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:20:17.023587    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcce1be48_6584_418d_a882_c30ad29e4f22.slice/crio-660fa5aa72eb34e12402d4386b5abed30948a2c83e8b6d240443b0645896f4a2 WatchSource:0}: Error finding container 660fa5aa72eb34e12402d4386b5abed30948a2c83e8b6d240443b0645896f4a2: Status 404 returned error can't find the container with id 660fa5aa72eb34e12402d4386b5abed30948a2c83e8b6d240443b0645896f4a2
Sep 08 04:20:17.059428 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:17.059391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540384" event={"ID":"cce1be48-6584-418d-a882-c30ad29e4f22","Type":"ContainerStarted","Data":"660fa5aa72eb34e12402d4386b5abed30948a2c83e8b6d240443b0645896f4a2"}
Sep 08 04:20:17.060679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:17.060660    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:17.060679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:17.060675    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e94fe899e2d6719bf034fcdaee4335425a1352393840ead5e1d84b4be2f6b96"
Sep 08 04:20:17.072895 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:17.072861    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7840-140093167540384" oldPodUID="9e370acc-ac32-4b41-a539-ba7e99062275" podUID="cce1be48-6584-418d-a882-c30ad29e4f22"
Sep 08 04:20:18.159620 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:18.159593    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e370acc-ac32-4b41-a539-ba7e99062275" path="/var/lib/kubelet/pods/9e370acc-ac32-4b41-a539-ba7e99062275/volumes"
Sep 08 04:20:19.071092 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:19.071059    2583 generic.go:356] "Generic (PLEG): container finished" podID="cce1be48-6584-418d-a882-c30ad29e4f22" containerID="ff48df36b064699af47bbae64c0ab8952909b6c62cf2a224a6504b212c4d2316" exitCode=0
Sep 08 04:20:19.071252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:19.071146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540384" event={"ID":"cce1be48-6584-418d-a882-c30ad29e4f22","Type":"ContainerDied","Data":"ff48df36b064699af47bbae64c0ab8952909b6c62cf2a224a6504b212c4d2316"}
Sep 08 04:20:20.169816 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.169732    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:20:20.170205 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.170159    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:20:20.211380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.211360    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:20.236598 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.236573    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7840-140093167540384_cce1be48-6584-418d-a882-c30ad29e4f22/test-curl-7840-140093167540384/0.log"
Sep 08 04:20:20.241355 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.241336    2583 scope.go:117] "RemoveContainer" containerID="2dd017a064476c9738796d298329758b86e99f142b13dc06c1b430a71f305145"
Sep 08 04:20:20.266992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.266964    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:20.271488 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.271464    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7840-140093167540384"]
Sep 08 04:20:20.314226 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.314205    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gzkg\" (UniqueName: \"kubernetes.io/projected/cce1be48-6584-418d-a882-c30ad29e4f22-kube-api-access-6gzkg\") pod \"cce1be48-6584-418d-a882-c30ad29e4f22\" (UID: \"cce1be48-6584-418d-a882-c30ad29e4f22\") "
Sep 08 04:20:20.316200 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.316172    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce1be48-6584-418d-a882-c30ad29e4f22-kube-api-access-6gzkg" (OuterVolumeSpecName: "kube-api-access-6gzkg") pod "cce1be48-6584-418d-a882-c30ad29e4f22" (UID: "cce1be48-6584-418d-a882-c30ad29e4f22"). InnerVolumeSpecName "kube-api-access-6gzkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:20:20.391015 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.390944    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:20.391338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.391320    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce1be48-6584-418d-a882-c30ad29e4f22" containerName="test-curl-7840-140093167540384"
Sep 08 04:20:20.391338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.391338    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce1be48-6584-418d-a882-c30ad29e4f22" containerName="test-curl-7840-140093167540384"
Sep 08 04:20:20.391460 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.391416    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce1be48-6584-418d-a882-c30ad29e4f22" containerName="test-curl-7840-140093167540384"
Sep 08 04:20:20.395461 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.395443    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:20.401891 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.401867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:20.415414 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.415390    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzkg\" (UniqueName: \"kubernetes.io/projected/cce1be48-6584-418d-a882-c30ad29e4f22-kube-api-access-6gzkg\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:20:20.516188 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.516168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8pk\" (UniqueName: \"kubernetes.io/projected/a6303a03-413d-46a1-8ac2-f2945e46b367-kube-api-access-7r8pk\") pod \"test-curl-7840-140093167540960\" (UID: \"a6303a03-413d-46a1-8ac2-f2945e46b367\") " pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:20.616988 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.616963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8pk\" (UniqueName: \"kubernetes.io/projected/a6303a03-413d-46a1-8ac2-f2945e46b367-kube-api-access-7r8pk\") pod \"test-curl-7840-140093167540960\" (UID: \"a6303a03-413d-46a1-8ac2-f2945e46b367\") " pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:20.625221 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.625200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8pk\" (UniqueName: \"kubernetes.io/projected/a6303a03-413d-46a1-8ac2-f2945e46b367-kube-api-access-7r8pk\") pod \"test-curl-7840-140093167540960\" (UID: \"a6303a03-413d-46a1-8ac2-f2945e46b367\") " pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:20.706355 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.706292    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:20.976568 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:20.976497    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:20.979157 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:20:20.979130    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6303a03_413d_46a1_8ac2_f2945e46b367.slice/crio-866a100d46e56acd916768e0a32f22541c6ffe099de4b6db56baec961325bbf2 WatchSource:0}: Error finding container 866a100d46e56acd916768e0a32f22541c6ffe099de4b6db56baec961325bbf2: Status 404 returned error can't find the container with id 866a100d46e56acd916768e0a32f22541c6ffe099de4b6db56baec961325bbf2
Sep 08 04:20:21.080721 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:21.080701    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:21.080842 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:21.080753    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660fa5aa72eb34e12402d4386b5abed30948a2c83e8b6d240443b0645896f4a2"
Sep 08 04:20:21.081986 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:21.081962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540960" event={"ID":"a6303a03-413d-46a1-8ac2-f2945e46b367","Type":"ContainerStarted","Data":"866a100d46e56acd916768e0a32f22541c6ffe099de4b6db56baec961325bbf2"}
Sep 08 04:20:21.091458 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:20:21.091432    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7840-140093167540384\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="cce1be48-6584-418d-a882-c30ad29e4f22" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:22.088279 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:22.088243    2583 generic.go:356] "Generic (PLEG): container finished" podID="a6303a03-413d-46a1-8ac2-f2945e46b367" containerID="8d647d7306f69ac21e0f5eb0ef318a80432f5c76c68e0ea28aaadcf9f306b821" exitCode=0
Sep 08 04:20:22.088656 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:22.088308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540960" event={"ID":"a6303a03-413d-46a1-8ac2-f2945e46b367","Type":"ContainerDied","Data":"8d647d7306f69ac21e0f5eb0ef318a80432f5c76c68e0ea28aaadcf9f306b821"}
Sep 08 04:20:22.109343 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:20:22.109308    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7840-140093167540384\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="cce1be48-6584-418d-a882-c30ad29e4f22" pod="opendatahub/test-curl-7840-140093167540384"
Sep 08 04:20:22.159536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:22.159510    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce1be48-6584-418d-a882-c30ad29e4f22" path="/var/lib/kubelet/pods/cce1be48-6584-418d-a882-c30ad29e4f22/volumes"
Sep 08 04:20:23.222362 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.222336    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:23.253960 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.253933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7840-140093167540960_a6303a03-413d-46a1-8ac2-f2945e46b367/test-curl-7840-140093167540960/0.log"
Sep 08 04:20:23.276840 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.276815    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:23.280624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.280602    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:23.340047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.340021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r8pk\" (UniqueName: \"kubernetes.io/projected/a6303a03-413d-46a1-8ac2-f2945e46b367-kube-api-access-7r8pk\") pod \"a6303a03-413d-46a1-8ac2-f2945e46b367\" (UID: \"a6303a03-413d-46a1-8ac2-f2945e46b367\") "
Sep 08 04:20:23.342046 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.342023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6303a03-413d-46a1-8ac2-f2945e46b367-kube-api-access-7r8pk" (OuterVolumeSpecName: "kube-api-access-7r8pk") pod "a6303a03-413d-46a1-8ac2-f2945e46b367" (UID: "a6303a03-413d-46a1-8ac2-f2945e46b367"). InnerVolumeSpecName "kube-api-access-7r8pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:20:23.414528 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.414500    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:23.414842 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.414829    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6303a03-413d-46a1-8ac2-f2945e46b367" containerName="test-curl-7840-140093167540960"
Sep 08 04:20:23.414896 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.414843    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6303a03-413d-46a1-8ac2-f2945e46b367" containerName="test-curl-7840-140093167540960"
Sep 08 04:20:23.414935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.414904    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6303a03-413d-46a1-8ac2-f2945e46b367" containerName="test-curl-7840-140093167540960"
Sep 08 04:20:23.417587 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.417572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:23.428356 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.428331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:23.440829 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.440809    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r8pk\" (UniqueName: \"kubernetes.io/projected/a6303a03-413d-46a1-8ac2-f2945e46b367-kube-api-access-7r8pk\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:20:23.542107 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.542080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97rn\" (UniqueName: \"kubernetes.io/projected/85bdc666-51ab-4ca0-913a-cec8f16a4465-kube-api-access-x97rn\") pod \"test-curl-7840-140093167540960\" (UID: \"85bdc666-51ab-4ca0-913a-cec8f16a4465\") " pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:23.643312 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.643252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97rn\" (UniqueName: \"kubernetes.io/projected/85bdc666-51ab-4ca0-913a-cec8f16a4465-kube-api-access-x97rn\") pod \"test-curl-7840-140093167540960\" (UID: \"85bdc666-51ab-4ca0-913a-cec8f16a4465\") " pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:23.654511 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.654489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97rn\" (UniqueName: \"kubernetes.io/projected/85bdc666-51ab-4ca0-913a-cec8f16a4465-kube-api-access-x97rn\") pod \"test-curl-7840-140093167540960\" (UID: \"85bdc666-51ab-4ca0-913a-cec8f16a4465\") " pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:23.726293 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.726261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:23.995932 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:23.995901    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:23.997590 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:20:23.997562    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85bdc666_51ab_4ca0_913a_cec8f16a4465.slice/crio-d02d57a1d2e32cfd555c02c3b75e963f689e876ed331bb8cb9d46294c0719bfe WatchSource:0}: Error finding container d02d57a1d2e32cfd555c02c3b75e963f689e876ed331bb8cb9d46294c0719bfe: Status 404 returned error can't find the container with id d02d57a1d2e32cfd555c02c3b75e963f689e876ed331bb8cb9d46294c0719bfe
Sep 08 04:20:24.097916 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:24.097893    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:24.098022 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:24.097893    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866a100d46e56acd916768e0a32f22541c6ffe099de4b6db56baec961325bbf2"
Sep 08 04:20:24.099200 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:24.099174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540960" event={"ID":"85bdc666-51ab-4ca0-913a-cec8f16a4465","Type":"ContainerStarted","Data":"d02d57a1d2e32cfd555c02c3b75e963f689e876ed331bb8cb9d46294c0719bfe"}
Sep 08 04:20:24.110143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:24.110122    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7840-140093167540960" oldPodUID="a6303a03-413d-46a1-8ac2-f2945e46b367" podUID="85bdc666-51ab-4ca0-913a-cec8f16a4465"
Sep 08 04:20:24.159894 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:24.159871    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6303a03-413d-46a1-8ac2-f2945e46b367" path="/var/lib/kubelet/pods/a6303a03-413d-46a1-8ac2-f2945e46b367/volumes"
Sep 08 04:20:25.106120 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:25.106090    2583 generic.go:356] "Generic (PLEG): container finished" podID="85bdc666-51ab-4ca0-913a-cec8f16a4465" containerID="c3b6faebfa6d767fc7a69539e3b58105123cb553c91fe23bb100480cf7eef827" exitCode=0
Sep 08 04:20:25.106510 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:25.106174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-140093167540960" event={"ID":"85bdc666-51ab-4ca0-913a-cec8f16a4465","Type":"ContainerDied","Data":"c3b6faebfa6d767fc7a69539e3b58105123cb553c91fe23bb100480cf7eef827"}
Sep 08 04:20:26.242247 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.242223    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:26.313347 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.313318    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7840-140093167540960_85bdc666-51ab-4ca0-913a-cec8f16a4465/test-curl-7840-140093167540960/0.log"
Sep 08 04:20:26.355015 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.354984    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:26.363650 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.363588    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x97rn\" (UniqueName: \"kubernetes.io/projected/85bdc666-51ab-4ca0-913a-cec8f16a4465-kube-api-access-x97rn\") pod \"85bdc666-51ab-4ca0-913a-cec8f16a4465\" (UID: \"85bdc666-51ab-4ca0-913a-cec8f16a4465\") "
Sep 08 04:20:26.365623 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.365599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bdc666-51ab-4ca0-913a-cec8f16a4465-kube-api-access-x97rn" (OuterVolumeSpecName: "kube-api-access-x97rn") pod "85bdc666-51ab-4ca0-913a-cec8f16a4465" (UID: "85bdc666-51ab-4ca0-913a-cec8f16a4465"). InnerVolumeSpecName "kube-api-access-x97rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:20:26.383977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.383953    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7840-140093167540960"]
Sep 08 04:20:26.464389 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:26.464367    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97rn\" (UniqueName: \"kubernetes.io/projected/85bdc666-51ab-4ca0-913a-cec8f16a4465-kube-api-access-x97rn\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:20:27.117380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:27.117350    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:27.117380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:27.117375    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d02d57a1d2e32cfd555c02c3b75e963f689e876ed331bb8cb9d46294c0719bfe"
Sep 08 04:20:27.127299 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:20:27.127264    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7840-140093167540960\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="85bdc666-51ab-4ca0-913a-cec8f16a4465" pod="opendatahub/test-curl-7840-140093167540960"
Sep 08 04:20:28.159708 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:28.159675    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85bdc666-51ab-4ca0-913a-cec8f16a4465" path="/var/lib/kubelet/pods/85bdc666-51ab-4ca0-913a-cec8f16a4465/volumes"
Sep 08 04:20:31.547119 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.547088    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"]
Sep 08 04:20:31.547580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.547526    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bdc666-51ab-4ca0-913a-cec8f16a4465" containerName="test-curl-7840-140093167540960"
Sep 08 04:20:31.547650 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.547592    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bdc666-51ab-4ca0-913a-cec8f16a4465" containerName="test-curl-7840-140093167540960"
Sep 08 04:20:31.547716 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.547697    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bdc666-51ab-4ca0-913a-cec8f16a4465" containerName="test-curl-7840-140093167540960"
Sep 08 04:20:31.550775 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.550757    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.553267 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.553243    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-777fbd7c-serving-cert\""
Sep 08 04:20:31.564514 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.564489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"]
Sep 08 04:20:31.701665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.701549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpt9r\" (UniqueName: \"kubernetes.io/projected/932a12fa-62ee-48ab-8c4a-46725a65d141-kube-api-access-qpt9r\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.701838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.701719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-metrics-certs\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.701838 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.701823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-maas-api-tls\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.701941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.701865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932a12fa-62ee-48ab-8c4a-46725a65d141-openshift-service-ca\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.802719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.802651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpt9r\" (UniqueName: \"kubernetes.io/projected/932a12fa-62ee-48ab-8c4a-46725a65d141-kube-api-access-qpt9r\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.802719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.802694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-metrics-certs\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.802719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.802714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-maas-api-tls\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.802924 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.802767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932a12fa-62ee-48ab-8c4a-46725a65d141-openshift-service-ca\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.803484 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.803460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932a12fa-62ee-48ab-8c4a-46725a65d141-openshift-service-ca\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.805070 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.805048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-metrics-certs\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.805174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.805144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-maas-api-tls\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.814648 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.814627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpt9r\" (UniqueName: \"kubernetes.io/projected/932a12fa-62ee-48ab-8c4a-46725a65d141-kube-api-access-qpt9r\") pod \"maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:31.863519 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:31.863496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:32.137832 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.137704    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"]
Sep 08 04:20:32.140398 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:20:32.140369    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932a12fa_62ee_48ab_8c4a_46725a65d141.slice/crio-c0cebba2a160b18a6660f495b14b44b1fea01e94aa9b23f8229730d96c6b2965 WatchSource:0}: Error finding container c0cebba2a160b18a6660f495b14b44b1fea01e94aa9b23f8229730d96c6b2965: Status 404 returned error can't find the container with id c0cebba2a160b18a6660f495b14b44b1fea01e94aa9b23f8229730d96c6b2965
Sep 08 04:20:32.313191 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.313158    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"]
Sep 08 04:20:32.317726 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.317710    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.323936 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.323907    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-777fbd7c\""
Sep 08 04:20:32.324080 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.323990    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-777fbd7c-dockercfg-rdb6q\""
Sep 08 04:20:32.332959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.332936    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"]
Sep 08 04:20:32.408843 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.408773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-plugins-config-volume\") pod \"payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") " pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.408955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.408836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbk8\" (UniqueName: \"kubernetes.io/projected/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-kube-api-access-srbk8\") pod \"payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") " pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.510152 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.510128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-plugins-config-volume\") pod \"payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") " pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.510243 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.510176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbk8\" (UniqueName: \"kubernetes.io/projected/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-kube-api-access-srbk8\") pod \"payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") " pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.510776 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.510758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-plugins-config-volume\") pod \"payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") " pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.519614 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.519587    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbk8\" (UniqueName: \"kubernetes.io/projected/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-kube-api-access-srbk8\") pod \"payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") " pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.628122 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.628090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:32.909014 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:32.908928    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"]
Sep 08 04:20:32.911624 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:20:32.911603    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b07378a_85ab_47b7_82c5_b615c7f6c6ad.slice/crio-fa2e22fe8cec7bece4b0cd9e240ad028085bf5895ea1db93613d70cf3ac52eb5 WatchSource:0}: Error finding container fa2e22fe8cec7bece4b0cd9e240ad028085bf5895ea1db93613d70cf3ac52eb5: Status 404 returned error can't find the container with id fa2e22fe8cec7bece4b0cd9e240ad028085bf5895ea1db93613d70cf3ac52eb5
Sep 08 04:20:33.148224 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.148188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv" event={"ID":"5b07378a-85ab-47b7-82c5-b615c7f6c6ad","Type":"ContainerStarted","Data":"af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95"}
Sep 08 04:20:33.148419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.148251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv" event={"ID":"5b07378a-85ab-47b7-82c5-b615c7f6c6ad","Type":"ContainerStarted","Data":"fa2e22fe8cec7bece4b0cd9e240ad028085bf5895ea1db93613d70cf3ac52eb5"}
Sep 08 04:20:33.148419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.148285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:20:33.149903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.149878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw" event={"ID":"932a12fa-62ee-48ab-8c4a-46725a65d141","Type":"ContainerStarted","Data":"fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba"}
Sep 08 04:20:33.150033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.149916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw" event={"ID":"932a12fa-62ee-48ab-8c4a-46725a65d141","Type":"ContainerStarted","Data":"c0cebba2a160b18a6660f495b14b44b1fea01e94aa9b23f8229730d96c6b2965"}
Sep 08 04:20:33.150033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.149955    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:33.168088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.168047    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv" podStartSLOduration=1.168033785 podStartE2EDuration="1.168033785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:20:33.166142166 +0000 UTC m=+1213.524621738" watchObservedRunningTime="2026-09-08 04:20:33.168033785 +0000 UTC m=+1213.526513345"
Sep 08 04:20:33.200270 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:33.200195    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw" podStartSLOduration=1.85609747 podStartE2EDuration="2.200177909s" totalImagesPullingTime="344.080439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:20:33.198225571 +0000 UTC m=+1213.556705133" watchObservedRunningTime="2026-09-08 04:20:33.200177909 +0000 UTC m=+1213.558657474"
Sep 08 04:20:37.660725 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:37.660693    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:20:39.160149 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:39.160117    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:20:44.159943 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:20:44.159914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:21:04.646044 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.646004    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"]
Sep 08 04:21:04.649492 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.649473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.652374 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.652355    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"openshift-service-ca.crt\""
Sep 08 04:21:04.652374 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.652368    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"auth-test-model-c2a8bb-kserve-self-signed-certs\""
Sep 08 04:21:04.652546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.652427    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"kube-root-ca.crt\""
Sep 08 04:21:04.653053 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.653033    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"default-dockercfg-w6tcb\""
Sep 08 04:21:04.659520 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.659501    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"]
Sep 08 04:21:04.741140 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.741110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tls-certs\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.741240 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.741147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-dshm\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.741240 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.741171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4pv\" (UniqueName: \"kubernetes.io/projected/3ff1c94b-9438-4fa5-b2f7-a57336273d26-kube-api-access-6n4pv\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.741319 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.741267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tmp-dir\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.741319 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.741301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-home\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.741416 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.741329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-model-cache\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842567 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tmp-dir\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-home\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-model-cache\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842795 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tls-certs\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842855 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-dshm\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842855 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4pv\" (UniqueName: \"kubernetes.io/projected/3ff1c94b-9438-4fa5-b2f7-a57336273d26-kube-api-access-6n4pv\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.842969 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.842952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tmp-dir\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.843025 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.843015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-model-cache\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.843086 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.843051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-home\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.844890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.844865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-dshm\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.845276 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.845253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tls-certs\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.851717 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.851699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4pv\" (UniqueName: \"kubernetes.io/projected/3ff1c94b-9438-4fa5-b2f7-a57336273d26-kube-api-access-6n4pv\") pod \"auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:04.959146 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:04.959096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:05.229204 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:05.229181    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"]
Sep 08 04:21:05.231689 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:21:05.231661    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ff1c94b_9438_4fa5_b2f7_a57336273d26.slice/crio-2dbfe3e84900872faec13e3de8c5b0225b391f0abb6d6d836a92462a3926de97 WatchSource:0}: Error finding container 2dbfe3e84900872faec13e3de8c5b0225b391f0abb6d6d836a92462a3926de97: Status 404 returned error can't find the container with id 2dbfe3e84900872faec13e3de8c5b0225b391f0abb6d6d836a92462a3926de97
Sep 08 04:21:05.288545 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:05.288517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4" event={"ID":"3ff1c94b-9438-4fa5-b2f7-a57336273d26","Type":"ContainerStarted","Data":"2dbfe3e84900872faec13e3de8c5b0225b391f0abb6d6d836a92462a3926de97"}
Sep 08 04:21:06.294696 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:06.294661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4" event={"ID":"3ff1c94b-9438-4fa5-b2f7-a57336273d26","Type":"ContainerStarted","Data":"084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147"}
Sep 08 04:21:06.313475 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:06.313423    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4" podStartSLOduration=2.313408692 podStartE2EDuration="2.313408692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:21:06.311107348 +0000 UTC m=+1246.669586909" watchObservedRunningTime="2026-09-08 04:21:06.313408692 +0000 UTC m=+1246.671888251"
Sep 08 04:21:10.556997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:10.556954    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:21:14.959659 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:14.959591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:14.959659 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:14.959642    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:14.971575 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:14.971552    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:15.347152 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:15.347125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:21:20.260934 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:21:20.260905    2583 scope.go:117] "RemoveContainer" containerID="62e803dee80ee5fbdabb61ef0b1d01c0ff54b5e393d084b815d97a5c4b047aa6"
Sep 08 04:22:05.751586 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:05.751541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:22:13.348538 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.348503    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"]
Sep 08 04:22:13.352095 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.352071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.355312 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.355286    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"openshift-service-ca.crt\""
Sep 08 04:22:13.355406 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.355287    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"kube-root-ca.crt\""
Sep 08 04:22:13.355475 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.355293    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"auth-test-model-720789-kserve-self-signed-certs\""
Sep 08 04:22:13.355475 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.355409    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"default-dockercfg-w5qg5\""
Sep 08 04:22:13.360826 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.360804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"]
Sep 08 04:22:13.440644 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.440621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-dshm\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.440771 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.440649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxftl\" (UniqueName: \"kubernetes.io/projected/65117aea-1fb1-42d2-af73-36de00aab3bd-kube-api-access-cxftl\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.440771 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.440681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65117aea-1fb1-42d2-af73-36de00aab3bd-tls-certs\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.440771 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.440735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-tmp-dir\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.440916 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.440824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-home\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.440916 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.440842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-model-cache\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541362 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-home\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541457 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-model-cache\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541457 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-dshm\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541457 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxftl\" (UniqueName: \"kubernetes.io/projected/65117aea-1fb1-42d2-af73-36de00aab3bd-kube-api-access-cxftl\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541457 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65117aea-1fb1-42d2-af73-36de00aab3bd-tls-certs\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541672 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-tmp-dir\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541764 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-model-cache\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541826 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-home\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.541925 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.541908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-tmp-dir\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.543597 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.543577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-dshm\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.543781 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.543766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65117aea-1fb1-42d2-af73-36de00aab3bd-tls-certs\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.550378 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.550359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxftl\" (UniqueName: \"kubernetes.io/projected/65117aea-1fb1-42d2-af73-36de00aab3bd-kube-api-access-cxftl\") pod \"auth-test-model-720789-kserve-7df9d86855-6sjmp\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") " pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.661799 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.661716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:13.929356 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:13.929328    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"]
Sep 08 04:22:13.931282 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:22:13.931256    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65117aea_1fb1_42d2_af73_36de00aab3bd.slice/crio-e60ea1ea2af7857187bbf08ef38900199cccc19c3ed502240303044f1aeb8640 WatchSource:0}: Error finding container e60ea1ea2af7857187bbf08ef38900199cccc19c3ed502240303044f1aeb8640: Status 404 returned error can't find the container with id e60ea1ea2af7857187bbf08ef38900199cccc19c3ed502240303044f1aeb8640
Sep 08 04:22:14.608364 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:14.608331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp" event={"ID":"65117aea-1fb1-42d2-af73-36de00aab3bd","Type":"ContainerStarted","Data":"3c49c0654e7654688e596f66a4948fd1cea16e5b15807a2a6f7f70b464729f3e"}
Sep 08 04:22:14.608717 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:14.608369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp" event={"ID":"65117aea-1fb1-42d2-af73-36de00aab3bd","Type":"ContainerStarted","Data":"e60ea1ea2af7857187bbf08ef38900199cccc19c3ed502240303044f1aeb8640"}
Sep 08 04:22:14.626088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:14.626040    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp" podStartSLOduration=1.626026037 podStartE2EDuration="1.626026037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:22:14.624538737 +0000 UTC m=+1314.983018298" watchObservedRunningTime="2026-09-08 04:22:14.626026037 +0000 UTC m=+1314.984505600"
Sep 08 04:22:18.952453 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:18.952397    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:22:23.662546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:23.662514    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:23.662914 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:23.662558    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:23.674660 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:23.674636    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:24.662787 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:24.662762    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:22:41.161216 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.161184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:22:41.401794 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.401764    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"]
Sep 08 04:22:41.401971 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.401944    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw" podUID="932a12fa-62ee-48ab-8c4a-46725a65d141" containerName="maas-api" containerID="cri-o://fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba" gracePeriod=30
Sep 08 04:22:41.450110 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.450080    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"]
Sep 08 04:22:41.450320 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.450296    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv" podUID="5b07378a-85ab-47b7-82c5-b615c7f6c6ad" containerName="payload-processing" containerID="cri-o://af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95" gracePeriod=30
Sep 08 04:22:41.698799 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.698776    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:22:41.720002 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.719977    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:22:41.724643 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.724620    2583 generic.go:356] "Generic (PLEG): container finished" podID="5b07378a-85ab-47b7-82c5-b615c7f6c6ad" containerID="af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95" exitCode=0
Sep 08 04:22:41.724924 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.724815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv" event={"ID":"5b07378a-85ab-47b7-82c5-b615c7f6c6ad","Type":"ContainerDied","Data":"af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95"}
Sep 08 04:22:41.724924 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.724864    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"
Sep 08 04:22:41.724924 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.724877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv" event={"ID":"5b07378a-85ab-47b7-82c5-b615c7f6c6ad","Type":"ContainerDied","Data":"fa2e22fe8cec7bece4b0cd9e240ad028085bf5895ea1db93613d70cf3ac52eb5"}
Sep 08 04:22:41.724924 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.724907    2583 scope.go:117] "RemoveContainer" containerID="af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95"
Sep 08 04:22:41.726339 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.726308    2583 generic.go:356] "Generic (PLEG): container finished" podID="932a12fa-62ee-48ab-8c4a-46725a65d141" containerID="fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba" exitCode=0
Sep 08 04:22:41.726463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.726345    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"
Sep 08 04:22:41.726463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.726349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw" event={"ID":"932a12fa-62ee-48ab-8c4a-46725a65d141","Type":"ContainerDied","Data":"fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba"}
Sep 08 04:22:41.726551 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.726464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw" event={"ID":"932a12fa-62ee-48ab-8c4a-46725a65d141","Type":"ContainerDied","Data":"c0cebba2a160b18a6660f495b14b44b1fea01e94aa9b23f8229730d96c6b2965"}
Sep 08 04:22:41.733852 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.733833    2583 scope.go:117] "RemoveContainer" containerID="af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95"
Sep 08 04:22:41.734134 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:22:41.734117    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95\": container with ID starting with af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95 not found: ID does not exist" containerID="af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95"
Sep 08 04:22:41.734207 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.734143    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95"} err="failed to get container status \"af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95\": rpc error: code = NotFound desc = could not find container \"af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95\": container with ID starting with af888e3c389a0062b88586ebbbd20635b725e21e0a5856eced0f575e0f453d95 not found: ID does not exist"
Sep 08 04:22:41.734207 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.734159    2583 scope.go:117] "RemoveContainer" containerID="fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba"
Sep 08 04:22:41.743287 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.743263    2583 scope.go:117] "RemoveContainer" containerID="fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba"
Sep 08 04:22:41.743900 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:22:41.743839    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba\": container with ID starting with fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba not found: ID does not exist" containerID="fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba"
Sep 08 04:22:41.743900 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.743874    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba"} err="failed to get container status \"fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba\": rpc error: code = NotFound desc = could not find container \"fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba\": container with ID starting with fbd875b2f198e70ae0a970f3469b9eca28bb07a74b28c34b7c363df901781aba not found: ID does not exist"
Sep 08 04:22:41.745221 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.745193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-metrics-certs\") pod \"932a12fa-62ee-48ab-8c4a-46725a65d141\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") "
Sep 08 04:22:41.745322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.745247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932a12fa-62ee-48ab-8c4a-46725a65d141-openshift-service-ca\") pod \"932a12fa-62ee-48ab-8c4a-46725a65d141\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") "
Sep 08 04:22:41.745322 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.745288    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpt9r\" (UniqueName: \"kubernetes.io/projected/932a12fa-62ee-48ab-8c4a-46725a65d141-kube-api-access-qpt9r\") pod \"932a12fa-62ee-48ab-8c4a-46725a65d141\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") "
Sep 08 04:22:41.745433 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.745326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-maas-api-tls\") pod \"932a12fa-62ee-48ab-8c4a-46725a65d141\" (UID: \"932a12fa-62ee-48ab-8c4a-46725a65d141\") "
Sep 08 04:22:41.745433 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.745379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srbk8\" (UniqueName: \"kubernetes.io/projected/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-kube-api-access-srbk8\") pod \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") "
Sep 08 04:22:41.745433 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.745409    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-plugins-config-volume\") pod \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\" (UID: \"5b07378a-85ab-47b7-82c5-b615c7f6c6ad\") "
Sep 08 04:22:41.746173 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.746021    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932a12fa-62ee-48ab-8c4a-46725a65d141-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "932a12fa-62ee-48ab-8c4a-46725a65d141" (UID: "932a12fa-62ee-48ab-8c4a-46725a65d141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:22:41.746173 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.746058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b07378a-85ab-47b7-82c5-b615c7f6c6ad" (UID: "5b07378a-85ab-47b7-82c5-b615c7f6c6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:22:41.747703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.747676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "932a12fa-62ee-48ab-8c4a-46725a65d141" (UID: "932a12fa-62ee-48ab-8c4a-46725a65d141"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:22:41.747998 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.747979    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "932a12fa-62ee-48ab-8c4a-46725a65d141" (UID: "932a12fa-62ee-48ab-8c4a-46725a65d141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:22:41.748166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.748151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932a12fa-62ee-48ab-8c4a-46725a65d141-kube-api-access-qpt9r" (OuterVolumeSpecName: "kube-api-access-qpt9r") pod "932a12fa-62ee-48ab-8c4a-46725a65d141" (UID: "932a12fa-62ee-48ab-8c4a-46725a65d141"). InnerVolumeSpecName "kube-api-access-qpt9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:22:41.748423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.748401    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-kube-api-access-srbk8" (OuterVolumeSpecName: "kube-api-access-srbk8") pod "5b07378a-85ab-47b7-82c5-b615c7f6c6ad" (UID: "5b07378a-85ab-47b7-82c5-b615c7f6c6ad"). InnerVolumeSpecName "kube-api-access-srbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:22:41.846768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.846749    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:22:41.846768 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.846770    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/932a12fa-62ee-48ab-8c4a-46725a65d141-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:22:41.846890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.846780    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpt9r\" (UniqueName: \"kubernetes.io/projected/932a12fa-62ee-48ab-8c4a-46725a65d141-kube-api-access-qpt9r\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:22:41.846890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.846788    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/932a12fa-62ee-48ab-8c4a-46725a65d141-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:22:41.846890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.846796    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbk8\" (UniqueName: \"kubernetes.io/projected/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-kube-api-access-srbk8\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:22:41.846890 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:41.846803    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b07378a-85ab-47b7-82c5-b615c7f6c6ad-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:22:42.047991 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:42.047969    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"]
Sep 08 04:22:42.051881 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:42.051861    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-777fbd7c-84d6f56c-kpnrw"]
Sep 08 04:22:42.062056 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:42.062037    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"]
Sep 08 04:22:42.065608 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:42.065583    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-777fbd7c-56445b5f77-h6bpv"]
Sep 08 04:22:42.159140 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:42.159117    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b07378a-85ab-47b7-82c5-b615c7f6c6ad" path="/var/lib/kubelet/pods/5b07378a-85ab-47b7-82c5-b615c7f6c6ad/volumes"
Sep 08 04:22:42.159473 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:42.159460    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932a12fa-62ee-48ab-8c4a-46725a65d141" path="/var/lib/kubelet/pods/932a12fa-62ee-48ab-8c4a-46725a65d141/volumes"
Sep 08 04:22:57.846711 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:22:57.846673    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:23:13.747053 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:23:13.747022    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:23:25.849341 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:23:25.849308    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:23:42.758544 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:23:42.758509    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:23:58.357117 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:23:58.357083    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:24:13.747181 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:13.747106    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:24:29.363011 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:29.362973    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:24:31.581424 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581388    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7843-1fac19"]
Sep 08 04:24:31.581821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581690    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932a12fa-62ee-48ab-8c4a-46725a65d141" containerName="maas-api"
Sep 08 04:24:31.581821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581701    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="932a12fa-62ee-48ab-8c4a-46725a65d141" containerName="maas-api"
Sep 08 04:24:31.581821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581717    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b07378a-85ab-47b7-82c5-b615c7f6c6ad" containerName="payload-processing"
Sep 08 04:24:31.581821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581723    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b07378a-85ab-47b7-82c5-b615c7f6c6ad" containerName="payload-processing"
Sep 08 04:24:31.581821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581798    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="932a12fa-62ee-48ab-8c4a-46725a65d141" containerName="maas-api"
Sep 08 04:24:31.581821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.581810    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b07378a-85ab-47b7-82c5-b615c7f6c6ad" containerName="payload-processing"
Sep 08 04:24:31.584358 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.584342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:31.587143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.587123    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nwvtr\""
Sep 08 04:24:31.592830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.592806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7843-1fac19"]
Sep 08 04:24:31.637689 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.637664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87tf\" (UniqueName: \"kubernetes.io/projected/e483de77-5ade-477a-bd11-6986e30eee8e-kube-api-access-z87tf\") pod \"mt-curl-7843-1fac19\" (UID: \"e483de77-5ade-477a-bd11-6986e30eee8e\") " pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:31.738624 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.738600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87tf\" (UniqueName: \"kubernetes.io/projected/e483de77-5ade-477a-bd11-6986e30eee8e-kube-api-access-z87tf\") pod \"mt-curl-7843-1fac19\" (UID: \"e483de77-5ade-477a-bd11-6986e30eee8e\") " pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:31.749574 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.749554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87tf\" (UniqueName: \"kubernetes.io/projected/e483de77-5ade-477a-bd11-6986e30eee8e-kube-api-access-z87tf\") pod \"mt-curl-7843-1fac19\" (UID: \"e483de77-5ade-477a-bd11-6986e30eee8e\") " pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:31.892912 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:31.892863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:32.165219 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:32.165191    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7843-1fac19"]
Sep 08 04:24:32.167410 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:24:32.167381    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode483de77_5ade_477a_bd11_6986e30eee8e.slice/crio-47745f5a5ed288aa8782beea822b8c2e1fba5c8e9c4cf55a83e670d2d6b68e99 WatchSource:0}: Error finding container 47745f5a5ed288aa8782beea822b8c2e1fba5c8e9c4cf55a83e670d2d6b68e99: Status 404 returned error can't find the container with id 47745f5a5ed288aa8782beea822b8c2e1fba5c8e9c4cf55a83e670d2d6b68e99
Sep 08 04:24:32.168581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:32.168564    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 04:24:32.214665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:32.214640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7843-1fac19" event={"ID":"e483de77-5ade-477a-bd11-6986e30eee8e","Type":"ContainerStarted","Data":"47745f5a5ed288aa8782beea822b8c2e1fba5c8e9c4cf55a83e670d2d6b68e99"}
Sep 08 04:24:33.220948 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:33.220882    2583 generic.go:356] "Generic (PLEG): container finished" podID="e483de77-5ade-477a-bd11-6986e30eee8e" containerID="253ec82455519af66d848f98030177843e1f7fda71a1d393065205be1a956ffa" exitCode=0
Sep 08 04:24:33.220948 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:33.220920    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7843-1fac19" event={"ID":"e483de77-5ade-477a-bd11-6986e30eee8e","Type":"ContainerDied","Data":"253ec82455519af66d848f98030177843e1f7fda71a1d393065205be1a956ffa"}
Sep 08 04:24:34.348142 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.348108    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:34.380921 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.380897    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7843-1fac19_e483de77-5ade-477a-bd11-6986e30eee8e/mt-curl-7843-1fac19/0.log"
Sep 08 04:24:34.405477 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.405449    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7843-1fac19"]
Sep 08 04:24:34.411194 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.411174    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7843-1fac19"]
Sep 08 04:24:34.460079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.460055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z87tf\" (UniqueName: \"kubernetes.io/projected/e483de77-5ade-477a-bd11-6986e30eee8e-kube-api-access-z87tf\") pod \"e483de77-5ade-477a-bd11-6986e30eee8e\" (UID: \"e483de77-5ade-477a-bd11-6986e30eee8e\") "
Sep 08 04:24:34.462163 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.462135    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e483de77-5ade-477a-bd11-6986e30eee8e-kube-api-access-z87tf" (OuterVolumeSpecName: "kube-api-access-z87tf") pod "e483de77-5ade-477a-bd11-6986e30eee8e" (UID: "e483de77-5ade-477a-bd11-6986e30eee8e"). InnerVolumeSpecName "kube-api-access-z87tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:24:34.560724 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:34.560698    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87tf\" (UniqueName: \"kubernetes.io/projected/e483de77-5ade-477a-bd11-6986e30eee8e-kube-api-access-z87tf\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:35.237379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:35.237349    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47745f5a5ed288aa8782beea822b8c2e1fba5c8e9c4cf55a83e670d2d6b68e99"
Sep 08 04:24:35.237379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:35.237350    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:35.247050 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:24:35.247023    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7843-1fac19\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="e483de77-5ade-477a-bd11-6986e30eee8e" pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:35.940325 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:35.940296    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"]
Sep 08 04:24:35.943944 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:24:35.943918    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7843-1fac19\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="e483de77-5ade-477a-bd11-6986e30eee8e" pod="opendatahub/mt-curl-7843-1fac19"
Sep 08 04:24:36.159266 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:36.159236    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e483de77-5ade-477a-bd11-6986e30eee8e" path="/var/lib/kubelet/pods/e483de77-5ade-477a-bd11-6986e30eee8e/volumes"
Sep 08 04:24:37.145794 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:37.145765    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"]
Sep 08 04:24:48.146684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:48.146635    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:24:50.958836 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:50.958793    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4" podUID="3ff1c94b-9438-4fa5-b2f7-a57336273d26" containerName="main" containerID="cri-o://084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147" gracePeriod=45
Sep 08 04:24:51.214854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.214794    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:24:51.292956 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.292929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-model-cache\") pod \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") "
Sep 08 04:24:51.293100 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.292979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-dshm\") pod \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") "
Sep 08 04:24:51.293100 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.292997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-home\") pod \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") "
Sep 08 04:24:51.293100 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tmp-dir\") pod \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") "
Sep 08 04:24:51.293230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n4pv\" (UniqueName: \"kubernetes.io/projected/3ff1c94b-9438-4fa5-b2f7-a57336273d26-kube-api-access-6n4pv\") pod \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") "
Sep 08 04:24:51.293230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tls-certs\") pod \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\" (UID: \"3ff1c94b-9438-4fa5-b2f7-a57336273d26\") "
Sep 08 04:24:51.293230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ff1c94b-9438-4fa5-b2f7-a57336273d26" (UID: "3ff1c94b-9438-4fa5-b2f7-a57336273d26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:51.293379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293308    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-home" (OuterVolumeSpecName: "home") pod "3ff1c94b-9438-4fa5-b2f7-a57336273d26" (UID: "3ff1c94b-9438-4fa5-b2f7-a57336273d26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:51.293379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ff1c94b-9438-4fa5-b2f7-a57336273d26" (UID: "3ff1c94b-9438-4fa5-b2f7-a57336273d26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:51.293487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293437    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:51.293487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293455    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:51.293487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.293471    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:51.295021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.294997    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-dshm" (OuterVolumeSpecName: "dshm") pod "3ff1c94b-9438-4fa5-b2f7-a57336273d26" (UID: "3ff1c94b-9438-4fa5-b2f7-a57336273d26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:51.295125 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.295085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff1c94b-9438-4fa5-b2f7-a57336273d26-kube-api-access-6n4pv" (OuterVolumeSpecName: "kube-api-access-6n4pv") pod "3ff1c94b-9438-4fa5-b2f7-a57336273d26" (UID: "3ff1c94b-9438-4fa5-b2f7-a57336273d26"). InnerVolumeSpecName "kube-api-access-6n4pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:24:51.295217 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.295194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ff1c94b-9438-4fa5-b2f7-a57336273d26" (UID: "3ff1c94b-9438-4fa5-b2f7-a57336273d26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:24:51.309006 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.308978    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ff1c94b-9438-4fa5-b2f7-a57336273d26" containerID="084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147" exitCode=0
Sep 08 04:24:51.309104 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.309026    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"
Sep 08 04:24:51.309104 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.309026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4" event={"ID":"3ff1c94b-9438-4fa5-b2f7-a57336273d26","Type":"ContainerDied","Data":"084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147"}
Sep 08 04:24:51.309184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.309128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4" event={"ID":"3ff1c94b-9438-4fa5-b2f7-a57336273d26","Type":"ContainerDied","Data":"2dbfe3e84900872faec13e3de8c5b0225b391f0abb6d6d836a92462a3926de97"}
Sep 08 04:24:51.309184 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.309146    2583 scope.go:117] "RemoveContainer" containerID="084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147"
Sep 08 04:24:51.317771 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.317734    2583 scope.go:117] "RemoveContainer" containerID="084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147"
Sep 08 04:24:51.318038 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:24:51.318018    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147\": container with ID starting with 084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147 not found: ID does not exist" containerID="084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147"
Sep 08 04:24:51.318103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.318045    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147"} err="failed to get container status \"084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147\": rpc error: code = NotFound desc = could not find container \"084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147\": container with ID starting with 084f9e74e9d258f0fa2c25d6553f8d2d50671b7f2a0e8c8048e155850b654147 not found: ID does not exist"
Sep 08 04:24:51.332163 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.332139    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"]
Sep 08 04:24:51.337053 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.337033    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/auth-test-model-c2a8bb-kserve-ff54bf9ff-hn7n4"]
Sep 08 04:24:51.394085 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.394062    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4pv\" (UniqueName: \"kubernetes.io/projected/3ff1c94b-9438-4fa5-b2f7-a57336273d26-kube-api-access-6n4pv\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:51.394085 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.394082    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c94b-9438-4fa5-b2f7-a57336273d26-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:51.394211 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:51.394092    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c94b-9438-4fa5-b2f7-a57336273d26-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:52.160015 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.159981    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff1c94b-9438-4fa5-b2f7-a57336273d26" path="/var/lib/kubelet/pods/3ff1c94b-9438-4fa5-b2f7-a57336273d26/volumes"
Sep 08 04:24:52.164463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.164419    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp" podUID="65117aea-1fb1-42d2-af73-36de00aab3bd" containerName="main" containerID="cri-o://3c49c0654e7654688e596f66a4948fd1cea16e5b15807a2a6f7f70b464729f3e" gracePeriod=45
Sep 08 04:24:52.315379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.315347    2583 generic.go:356] "Generic (PLEG): container finished" podID="65117aea-1fb1-42d2-af73-36de00aab3bd" containerID="3c49c0654e7654688e596f66a4948fd1cea16e5b15807a2a6f7f70b464729f3e" exitCode=0
Sep 08 04:24:52.315530 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.315422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp" event={"ID":"65117aea-1fb1-42d2-af73-36de00aab3bd","Type":"ContainerDied","Data":"3c49c0654e7654688e596f66a4948fd1cea16e5b15807a2a6f7f70b464729f3e"}
Sep 08 04:24:52.410365 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.410313    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:24:52.504266 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504241    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-tmp-dir\") pod \"65117aea-1fb1-42d2-af73-36de00aab3bd\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") "
Sep 08 04:24:52.504377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxftl\" (UniqueName: \"kubernetes.io/projected/65117aea-1fb1-42d2-af73-36de00aab3bd-kube-api-access-cxftl\") pod \"65117aea-1fb1-42d2-af73-36de00aab3bd\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") "
Sep 08 04:24:52.504377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65117aea-1fb1-42d2-af73-36de00aab3bd-tls-certs\") pod \"65117aea-1fb1-42d2-af73-36de00aab3bd\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") "
Sep 08 04:24:52.504377 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-model-cache\") pod \"65117aea-1fb1-42d2-af73-36de00aab3bd\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") "
Sep 08 04:24:52.504515 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-dshm\") pod \"65117aea-1fb1-42d2-af73-36de00aab3bd\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") "
Sep 08 04:24:52.504515 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504425    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-home\") pod \"65117aea-1fb1-42d2-af73-36de00aab3bd\" (UID: \"65117aea-1fb1-42d2-af73-36de00aab3bd\") "
Sep 08 04:24:52.504615 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504528    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65117aea-1fb1-42d2-af73-36de00aab3bd" (UID: "65117aea-1fb1-42d2-af73-36de00aab3bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:52.504679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504621    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "65117aea-1fb1-42d2-af73-36de00aab3bd" (UID: "65117aea-1fb1-42d2-af73-36de00aab3bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:52.504679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504632    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:52.504819 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.504770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-home" (OuterVolumeSpecName: "home") pod "65117aea-1fb1-42d2-af73-36de00aab3bd" (UID: "65117aea-1fb1-42d2-af73-36de00aab3bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:52.506611 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.506585    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-dshm" (OuterVolumeSpecName: "dshm") pod "65117aea-1fb1-42d2-af73-36de00aab3bd" (UID: "65117aea-1fb1-42d2-af73-36de00aab3bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:24:52.506611 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.506594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65117aea-1fb1-42d2-af73-36de00aab3bd-kube-api-access-cxftl" (OuterVolumeSpecName: "kube-api-access-cxftl") pod "65117aea-1fb1-42d2-af73-36de00aab3bd" (UID: "65117aea-1fb1-42d2-af73-36de00aab3bd"). InnerVolumeSpecName "kube-api-access-cxftl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:24:52.506770 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.506633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65117aea-1fb1-42d2-af73-36de00aab3bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65117aea-1fb1-42d2-af73-36de00aab3bd" (UID: "65117aea-1fb1-42d2-af73-36de00aab3bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:24:52.605984 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.605952    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxftl\" (UniqueName: \"kubernetes.io/projected/65117aea-1fb1-42d2-af73-36de00aab3bd-kube-api-access-cxftl\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:52.605984 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.605979    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65117aea-1fb1-42d2-af73-36de00aab3bd-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:52.606138 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.605993    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:52.606138 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.606004    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:52.606138 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:52.606015    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65117aea-1fb1-42d2-af73-36de00aab3bd-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:24:53.323081 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:53.323040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp" event={"ID":"65117aea-1fb1-42d2-af73-36de00aab3bd","Type":"ContainerDied","Data":"e60ea1ea2af7857187bbf08ef38900199cccc19c3ed502240303044f1aeb8640"}
Sep 08 04:24:53.323081 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:53.323079    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"
Sep 08 04:24:53.323552 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:53.323108    2583 scope.go:117] "RemoveContainer" containerID="3c49c0654e7654688e596f66a4948fd1cea16e5b15807a2a6f7f70b464729f3e"
Sep 08 04:24:53.345997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:53.345973    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"]
Sep 08 04:24:53.351220 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:53.351199    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/auth-test-model-720789-kserve-7df9d86855-6sjmp"]
Sep 08 04:24:54.159145 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:54.159112    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65117aea-1fb1-42d2-af73-36de00aab3bd" path="/var/lib/kubelet/pods/65117aea-1fb1-42d2-af73-36de00aab3bd/volumes"
Sep 08 04:24:55.946599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946568    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"]
Sep 08 04:24:55.946992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946897    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e483de77-5ade-477a-bd11-6986e30eee8e" containerName="mt-curl-7843-1fac19"
Sep 08 04:24:55.946992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946909    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e483de77-5ade-477a-bd11-6986e30eee8e" containerName="mt-curl-7843-1fac19"
Sep 08 04:24:55.946992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946917    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65117aea-1fb1-42d2-af73-36de00aab3bd" containerName="main"
Sep 08 04:24:55.946992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946924    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65117aea-1fb1-42d2-af73-36de00aab3bd" containerName="main"
Sep 08 04:24:55.946992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946943    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff1c94b-9438-4fa5-b2f7-a57336273d26" containerName="main"
Sep 08 04:24:55.946992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.946949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff1c94b-9438-4fa5-b2f7-a57336273d26" containerName="main"
Sep 08 04:24:55.947227 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.947003    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e483de77-5ade-477a-bd11-6986e30eee8e" containerName="mt-curl-7843-1fac19"
Sep 08 04:24:55.947227 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.947012    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff1c94b-9438-4fa5-b2f7-a57336273d26" containerName="main"
Sep 08 04:24:55.947227 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.947020    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65117aea-1fb1-42d2-af73-36de00aab3bd" containerName="main"
Sep 08 04:24:55.951096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.951074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:55.954630 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.954606    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"sub-test-model-c2a8bb-kserve-self-signed-certs\""
Sep 08 04:24:55.954766 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.954612    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"openshift-service-ca.crt\""
Sep 08 04:24:55.954766 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.954657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"default-dockercfg-w6tcb\""
Sep 08 04:24:55.954766 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.954712    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"kube-root-ca.crt\""
Sep 08 04:24:55.960623 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:55.960599    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"]
Sep 08 04:24:56.036682 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.036644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-dshm\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.036682 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.036687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-model-cache\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.036926 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.036711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-home\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.036926 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.036728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tmp-dir\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.036926 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.036834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tls-certs\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.036926 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.036890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwth\" (UniqueName: \"kubernetes.io/projected/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-kube-api-access-9hwth\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-dshm\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-model-cache\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-home\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tmp-dir\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tls-certs\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwth\" (UniqueName: \"kubernetes.io/projected/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-kube-api-access-9hwth\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137833 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-model-cache\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-home\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.137933 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.137854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tmp-dir\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.139507 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.139487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-dshm\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.139957 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.139937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tls-certs\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.145609 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.145589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwth\" (UniqueName: \"kubernetes.io/projected/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-kube-api-access-9hwth\") pod \"sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.261047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.261006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:24:56.532729 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:56.532703    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"]
Sep 08 04:24:56.535042 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:24:56.535016    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fbf9a60_f0b2_4a6f_9396_392982dd06e5.slice/crio-802e3f9c3b3c53046237be6a0280584a98708d5bef75f87cdf37a8777dbb40de WatchSource:0}: Error finding container 802e3f9c3b3c53046237be6a0280584a98708d5bef75f87cdf37a8777dbb40de: Status 404 returned error can't find the container with id 802e3f9c3b3c53046237be6a0280584a98708d5bef75f87cdf37a8777dbb40de
Sep 08 04:24:57.343383 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:57.343348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj" event={"ID":"3fbf9a60-f0b2-4a6f-9396-392982dd06e5","Type":"ContainerStarted","Data":"c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73"}
Sep 08 04:24:57.343383 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:57.343388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj" event={"ID":"3fbf9a60-f0b2-4a6f-9396-392982dd06e5","Type":"ContainerStarted","Data":"802e3f9c3b3c53046237be6a0280584a98708d5bef75f87cdf37a8777dbb40de"}
Sep 08 04:24:57.361034 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:24:57.360983    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj" podStartSLOduration=2.360970002 podStartE2EDuration="2.360970002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:24:57.359959043 +0000 UTC m=+1477.718438602" watchObservedRunningTime="2026-09-08 04:24:57.360970002 +0000 UTC m=+1477.719449559"
Sep 08 04:25:02.055667 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.055625    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"]
Sep 08 04:25:02.059177 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.059154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.061617 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.061593    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7bac07-serving-cert\""
Sep 08 04:25:02.069135 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.069070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"]
Sep 08 04:25:02.174981 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.174921    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:25:02.189775 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.189731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-metrics-certs\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.189911 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.189850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-maas-api-tls\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.189991 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.189971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db889c0-d7a0-4c11-b758-2608ccc0ac78-openshift-service-ca\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.190032 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.190020    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n85\" (UniqueName: \"kubernetes.io/projected/4db889c0-d7a0-4c11-b758-2608ccc0ac78-kube-api-access-g8n85\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.290830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.290794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db889c0-d7a0-4c11-b758-2608ccc0ac78-openshift-service-ca\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.290830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.290827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n85\" (UniqueName: \"kubernetes.io/projected/4db889c0-d7a0-4c11-b758-2608ccc0ac78-kube-api-access-g8n85\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.291077 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.290868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-metrics-certs\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.291077 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.290944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-maas-api-tls\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.291903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.291875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db889c0-d7a0-4c11-b758-2608ccc0ac78-openshift-service-ca\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.293536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.293513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-maas-api-tls\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.293725 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.293704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-metrics-certs\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.302894 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.302866    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n85\" (UniqueName: \"kubernetes.io/projected/4db889c0-d7a0-4c11-b758-2608ccc0ac78-kube-api-access-g8n85\") pod \"maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.372343 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.372276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:02.651947 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.651860    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"]
Sep 08 04:25:02.655302 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:25:02.655248    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db889c0_d7a0_4c11_b758_2608ccc0ac78.slice/crio-c505a51a6391e42bb4afae890d3abeb5d2675c464c2065217fb74813e6944332 WatchSource:0}: Error finding container c505a51a6391e42bb4afae890d3abeb5d2675c464c2065217fb74813e6944332: Status 404 returned error can't find the container with id c505a51a6391e42bb4afae890d3abeb5d2675c464c2065217fb74813e6944332
Sep 08 04:25:02.887910 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.883684    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"]
Sep 08 04:25:02.890352 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.890323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:02.893232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.893208    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7bac07-dockercfg-rkkh4\""
Sep 08 04:25:02.893338 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.893209    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7bac07\""
Sep 08 04:25:02.898575 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.898551    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"]
Sep 08 04:25:02.998625 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.998595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bee93d-9978-49a0-9bee-a90748023d12-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:02.998728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:02.998674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55jr\" (UniqueName: \"kubernetes.io/projected/d2bee93d-9978-49a0-9bee-a90748023d12-kube-api-access-z55jr\") pod \"payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:03.099649 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.099593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bee93d-9978-49a0-9bee-a90748023d12-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:03.100031 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.099706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55jr\" (UniqueName: \"kubernetes.io/projected/d2bee93d-9978-49a0-9bee-a90748023d12-kube-api-access-z55jr\") pod \"payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:03.100311 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.100292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bee93d-9978-49a0-9bee-a90748023d12-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:03.111992 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.111964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55jr\" (UniqueName: \"kubernetes.io/projected/d2bee93d-9978-49a0-9bee-a90748023d12-kube-api-access-z55jr\") pod \"payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:03.201729 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.201687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:03.370120 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.370072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5" event={"ID":"4db889c0-d7a0-4c11-b758-2608ccc0ac78","Type":"ContainerStarted","Data":"795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514"}
Sep 08 04:25:03.370278 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.370139    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:03.370278 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.370163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5" event={"ID":"4db889c0-d7a0-4c11-b758-2608ccc0ac78","Type":"ContainerStarted","Data":"c505a51a6391e42bb4afae890d3abeb5d2675c464c2065217fb74813e6944332"}
Sep 08 04:25:03.394926 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.394875    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5" podStartSLOduration=1.065851258 podStartE2EDuration="1.394859818s" totalImagesPullingTime="329.00856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:25:03.392641589 +0000 UTC m=+1483.751121147" watchObservedRunningTime="2026-09-08 04:25:03.394859818 +0000 UTC m=+1483.753339419"
Sep 08 04:25:03.483114 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:03.483088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"]
Sep 08 04:25:03.484249 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:25:03.484221    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bee93d_9978_49a0_9bee_a90748023d12.slice/crio-0b114330a79f3482d5fab524ae48d13c1d510396a1456f6a2f4e571329429a68 WatchSource:0}: Error finding container 0b114330a79f3482d5fab524ae48d13c1d510396a1456f6a2f4e571329429a68: Status 404 returned error can't find the container with id 0b114330a79f3482d5fab524ae48d13c1d510396a1456f6a2f4e571329429a68
Sep 08 04:25:04.377366 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:04.377327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd" event={"ID":"d2bee93d-9978-49a0-9bee-a90748023d12","Type":"ContainerStarted","Data":"74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502"}
Sep 08 04:25:04.377719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:04.377390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd" event={"ID":"d2bee93d-9978-49a0-9bee-a90748023d12","Type":"ContainerStarted","Data":"0b114330a79f3482d5fab524ae48d13c1d510396a1456f6a2f4e571329429a68"}
Sep 08 04:25:04.377719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:04.377415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:04.395163 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:04.395031    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd" podStartSLOduration=2.395012219 podStartE2EDuration="2.395012219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:25:04.394113654 +0000 UTC m=+1484.752593213" watchObservedRunningTime="2026-09-08 04:25:04.395012219 +0000 UTC m=+1484.753491782"
Sep 08 04:25:06.261758 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:06.261702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:25:06.261758 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:06.261766    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:25:06.274054 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:06.274031    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:25:06.398358 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:06.398333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:25:09.382818 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:09.382788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:25:15.384702 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:15.384670    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:25:20.199639 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:20.199611    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:25:20.200659 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:20.200637    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:25:20.550759 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:20.550714    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:25:37.450200 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.450168    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"]
Sep 08 04:25:37.453862 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.453840    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.457289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.457266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"sub-test-model-720789-kserve-self-signed-certs\""
Sep 08 04:25:37.457289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.457281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"openshift-service-ca.crt\""
Sep 08 04:25:37.457463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.457281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"kube-root-ca.crt\""
Sep 08 04:25:37.457463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.457319    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"default-dockercfg-w5qg5\""
Sep 08 04:25:37.466954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.466932    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"]
Sep 08 04:25:37.574789 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.574759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-model-cache\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.574903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.574796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-tmp-dir\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.574903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.574815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9641b082-71ba-4afe-a4f7-03e456f20ab5-tls-certs\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.574903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.574880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zj27\" (UniqueName: \"kubernetes.io/projected/9641b082-71ba-4afe-a4f7-03e456f20ab5-kube-api-access-8zj27\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.575010 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.574975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-home\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.575010 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.574995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-dshm\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675489 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj27\" (UniqueName: \"kubernetes.io/projected/9641b082-71ba-4afe-a4f7-03e456f20ab5-kube-api-access-8zj27\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675579 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-home\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675579 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-dshm\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675579 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-model-cache\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675691 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-tmp-dir\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675691 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9641b082-71ba-4afe-a4f7-03e456f20ab5-tls-certs\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.675994 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.675975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-model-cache\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.676057 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.676031    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-tmp-dir\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.676057 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.676042    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-home\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.677893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.677865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-dshm\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.678063 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.678045    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9641b082-71ba-4afe-a4f7-03e456f20ab5-tls-certs\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.684482 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.684462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj27\" (UniqueName: \"kubernetes.io/projected/9641b082-71ba-4afe-a4f7-03e456f20ab5-kube-api-access-8zj27\") pod \"sub-test-model-720789-kserve-bb55b55b5-gs5gn\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") " pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:37.763010 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:37.762991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:38.031818 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:38.031792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"]
Sep 08 04:25:38.034473 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:25:38.034449    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9641b082_71ba_4afe_a4f7_03e456f20ab5.slice/crio-0b46b2f8af97f32abfb1631ea036d20c27e40a65ee645c032a6cf684f05cfffb WatchSource:0}: Error finding container 0b46b2f8af97f32abfb1631ea036d20c27e40a65ee645c032a6cf684f05cfffb: Status 404 returned error can't find the container with id 0b46b2f8af97f32abfb1631ea036d20c27e40a65ee645c032a6cf684f05cfffb
Sep 08 04:25:38.532167 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:38.532134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn" event={"ID":"9641b082-71ba-4afe-a4f7-03e456f20ab5","Type":"ContainerStarted","Data":"c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1"}
Sep 08 04:25:38.532489 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:38.532184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn" event={"ID":"9641b082-71ba-4afe-a4f7-03e456f20ab5","Type":"ContainerStarted","Data":"0b46b2f8af97f32abfb1631ea036d20c27e40a65ee645c032a6cf684f05cfffb"}
Sep 08 04:25:38.548901 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:38.548858    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn" podStartSLOduration=1.54884392 podStartE2EDuration="1.54884392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:25:38.548505898 +0000 UTC m=+1518.906985456" watchObservedRunningTime="2026-09-08 04:25:38.54884392 +0000 UTC m=+1518.907323480"
Sep 08 04:25:45.459415 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:45.459318    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:25:47.763261 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:47.763230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:47.763716 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:47.763463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:47.775463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:47.775436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:25:48.589075 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:25:48.589051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:26:00.651577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:00.651541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:26:07.921706 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:07.921671    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"]
Sep 08 04:26:07.922118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:07.921885    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5" podUID="4db889c0-d7a0-4c11-b758-2608ccc0ac78" containerName="maas-api" containerID="cri-o://795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514" gracePeriod=30
Sep 08 04:26:07.992264 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:07.992232    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"]
Sep 08 04:26:07.992447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:07.992414    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd" podUID="d2bee93d-9978-49a0-9bee-a90748023d12" containerName="payload-pre-processing" containerID="cri-o://74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502" gracePeriod=30
Sep 08 04:26:08.232678 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.232655    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:26:08.263987 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.263962    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:26:08.314274 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.314245    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8n85\" (UniqueName: \"kubernetes.io/projected/4db889c0-d7a0-4c11-b758-2608ccc0ac78-kube-api-access-g8n85\") pod \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") "
Sep 08 04:26:08.314394 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.314336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-metrics-certs\") pod \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") "
Sep 08 04:26:08.314394 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.314373    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db889c0-d7a0-4c11-b758-2608ccc0ac78-openshift-service-ca\") pod \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") "
Sep 08 04:26:08.314507 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.314491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-maas-api-tls\") pod \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\" (UID: \"4db889c0-d7a0-4c11-b758-2608ccc0ac78\") "
Sep 08 04:26:08.314672 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.314650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db889c0-d7a0-4c11-b758-2608ccc0ac78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4db889c0-d7a0-4c11-b758-2608ccc0ac78" (UID: "4db889c0-d7a0-4c11-b758-2608ccc0ac78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:26:08.314877 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.314859    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db889c0-d7a0-4c11-b758-2608ccc0ac78-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:08.316250 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.316223    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db889c0-d7a0-4c11-b758-2608ccc0ac78-kube-api-access-g8n85" (OuterVolumeSpecName: "kube-api-access-g8n85") pod "4db889c0-d7a0-4c11-b758-2608ccc0ac78" (UID: "4db889c0-d7a0-4c11-b758-2608ccc0ac78"). InnerVolumeSpecName "kube-api-access-g8n85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:08.316342 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.316309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4db889c0-d7a0-4c11-b758-2608ccc0ac78" (UID: "4db889c0-d7a0-4c11-b758-2608ccc0ac78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:08.316399 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.316379    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4db889c0-d7a0-4c11-b758-2608ccc0ac78" (UID: "4db889c0-d7a0-4c11-b758-2608ccc0ac78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:08.415553 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.415532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bee93d-9978-49a0-9bee-a90748023d12-plugins-config-volume\") pod \"d2bee93d-9978-49a0-9bee-a90748023d12\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") "
Sep 08 04:26:08.415644 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.415568    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z55jr\" (UniqueName: \"kubernetes.io/projected/d2bee93d-9978-49a0-9bee-a90748023d12-kube-api-access-z55jr\") pod \"d2bee93d-9978-49a0-9bee-a90748023d12\" (UID: \"d2bee93d-9978-49a0-9bee-a90748023d12\") "
Sep 08 04:26:08.415734 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.415675    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:08.415734 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.415684    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4db889c0-d7a0-4c11-b758-2608ccc0ac78-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:08.415734 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.415692    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n85\" (UniqueName: \"kubernetes.io/projected/4db889c0-d7a0-4c11-b758-2608ccc0ac78-kube-api-access-g8n85\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:08.415942 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.415882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bee93d-9978-49a0-9bee-a90748023d12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2bee93d-9978-49a0-9bee-a90748023d12" (UID: "d2bee93d-9978-49a0-9bee-a90748023d12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:26:08.417232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.417211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bee93d-9978-49a0-9bee-a90748023d12-kube-api-access-z55jr" (OuterVolumeSpecName: "kube-api-access-z55jr") pod "d2bee93d-9978-49a0-9bee-a90748023d12" (UID: "d2bee93d-9978-49a0-9bee-a90748023d12"). InnerVolumeSpecName "kube-api-access-z55jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:08.516819 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.516799    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bee93d-9978-49a0-9bee-a90748023d12-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:08.516819 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.516818    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55jr\" (UniqueName: \"kubernetes.io/projected/d2bee93d-9978-49a0-9bee-a90748023d12-kube-api-access-z55jr\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:08.669965 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.669935    2583 generic.go:356] "Generic (PLEG): container finished" podID="4db889c0-d7a0-4c11-b758-2608ccc0ac78" containerID="795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514" exitCode=0
Sep 08 04:26:08.670039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.669979    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"
Sep 08 04:26:08.670039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.670024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5" event={"ID":"4db889c0-d7a0-4c11-b758-2608ccc0ac78","Type":"ContainerDied","Data":"795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514"}
Sep 08 04:26:08.670136 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.670075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5" event={"ID":"4db889c0-d7a0-4c11-b758-2608ccc0ac78","Type":"ContainerDied","Data":"c505a51a6391e42bb4afae890d3abeb5d2675c464c2065217fb74813e6944332"}
Sep 08 04:26:08.670136 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.670106    2583 scope.go:117] "RemoveContainer" containerID="795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514"
Sep 08 04:26:08.671471 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.671435    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2bee93d-9978-49a0-9bee-a90748023d12" containerID="74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502" exitCode=0
Sep 08 04:26:08.671584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.671469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd" event={"ID":"d2bee93d-9978-49a0-9bee-a90748023d12","Type":"ContainerDied","Data":"74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502"}
Sep 08 04:26:08.671584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.671515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd" event={"ID":"d2bee93d-9978-49a0-9bee-a90748023d12","Type":"ContainerDied","Data":"0b114330a79f3482d5fab524ae48d13c1d510396a1456f6a2f4e571329429a68"}
Sep 08 04:26:08.671584 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.671519    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"
Sep 08 04:26:08.680504 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.680486    2583 scope.go:117] "RemoveContainer" containerID="795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514"
Sep 08 04:26:08.680828 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:08.680811    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514\": container with ID starting with 795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514 not found: ID does not exist" containerID="795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514"
Sep 08 04:26:08.680883 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.680836    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514"} err="failed to get container status \"795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514\": rpc error: code = NotFound desc = could not find container \"795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514\": container with ID starting with 795ee9c9514c1046f7cf48c6c836c0d1b6e97dcee53b14449cce19385817f514 not found: ID does not exist"
Sep 08 04:26:08.680883 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.680853    2583 scope.go:117] "RemoveContainer" containerID="74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502"
Sep 08 04:26:08.688278 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.688256    2583 scope.go:117] "RemoveContainer" containerID="74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502"
Sep 08 04:26:08.688554 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:08.688518    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502\": container with ID starting with 74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502 not found: ID does not exist" containerID="74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502"
Sep 08 04:26:08.688606 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.688558    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502"} err="failed to get container status \"74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502\": rpc error: code = NotFound desc = could not find container \"74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502\": container with ID starting with 74243650658dd4b6f9681c7c8befdc172487e4b27d4b929447f7e6ea7ce31502 not found: ID does not exist"
Sep 08 04:26:08.695690 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.695667    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"]
Sep 08 04:26:08.701374 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.701354    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bac07-5cdb585cfb-vx9z5"]
Sep 08 04:26:08.713931 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.713910    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"]
Sep 08 04:26:08.730464 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:08.730445    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bac07-6dbc58c494-p5hsd"]
Sep 08 04:26:10.159582 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:10.159552    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db889c0-d7a0-4c11-b758-2608ccc0ac78" path="/var/lib/kubelet/pods/4db889c0-d7a0-4c11-b758-2608ccc0ac78/volumes"
Sep 08 04:26:10.159962 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:10.159907    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2bee93d-9978-49a0-9bee-a90748023d12" path="/var/lib/kubelet/pods/d2bee93d-9978-49a0-9bee-a90748023d12/volumes"
Sep 08 04:26:15.267384 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267353    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7843-9ad0b2"]
Sep 08 04:26:15.269526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267669    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2bee93d-9978-49a0-9bee-a90748023d12" containerName="payload-pre-processing"
Sep 08 04:26:15.269526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267680    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2bee93d-9978-49a0-9bee-a90748023d12" containerName="payload-pre-processing"
Sep 08 04:26:15.269526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267692    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db889c0-d7a0-4c11-b758-2608ccc0ac78" containerName="maas-api"
Sep 08 04:26:15.269526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267698    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db889c0-d7a0-4c11-b758-2608ccc0ac78" containerName="maas-api"
Sep 08 04:26:15.269526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267766    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2bee93d-9978-49a0-9bee-a90748023d12" containerName="payload-pre-processing"
Sep 08 04:26:15.269526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.267775    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db889c0-d7a0-4c11-b758-2608ccc0ac78" containerName="maas-api"
Sep 08 04:26:15.270378 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.270361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:15.273942 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.273921    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nwvtr\""
Sep 08 04:26:15.282977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.282957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7843-9ad0b2"]
Sep 08 04:26:15.366220 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.366193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7frw\" (UniqueName: \"kubernetes.io/projected/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225-kube-api-access-g7frw\") pod \"mt-curl-7843-9ad0b2\" (UID: \"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225\") " pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:15.466508 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.466483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7frw\" (UniqueName: \"kubernetes.io/projected/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225-kube-api-access-g7frw\") pod \"mt-curl-7843-9ad0b2\" (UID: \"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225\") " pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:15.475103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.475072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7frw\" (UniqueName: \"kubernetes.io/projected/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225-kube-api-access-g7frw\") pod \"mt-curl-7843-9ad0b2\" (UID: \"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225\") " pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:15.579463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.579405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:15.845752 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:15.845717    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7843-9ad0b2"]
Sep 08 04:26:15.847624 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:26:15.847597    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bbf84c9_b6c3_41a7_bd5e_fc4e3a962225.slice/crio-46106651b26d858c8e13b2ef07ac40fe173457fd238505d1568df53a9d88b146 WatchSource:0}: Error finding container 46106651b26d858c8e13b2ef07ac40fe173457fd238505d1568df53a9d88b146: Status 404 returned error can't find the container with id 46106651b26d858c8e13b2ef07ac40fe173457fd238505d1568df53a9d88b146
Sep 08 04:26:16.717345 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:16.717316    2583 generic.go:356] "Generic (PLEG): container finished" podID="8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225" containerID="0ac440e91306d28cc5030ec01afc7a711c069943c5bc87a1377bb4999872fe44" exitCode=0
Sep 08 04:26:16.717644 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:16.717353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7843-9ad0b2" event={"ID":"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225","Type":"ContainerDied","Data":"0ac440e91306d28cc5030ec01afc7a711c069943c5bc87a1377bb4999872fe44"}
Sep 08 04:26:16.717644 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:16.717385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7843-9ad0b2" event={"ID":"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225","Type":"ContainerStarted","Data":"46106651b26d858c8e13b2ef07ac40fe173457fd238505d1568df53a9d88b146"}
Sep 08 04:26:17.851030 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:17.851004    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:17.886849 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:17.886828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7843-9ad0b2_8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225/mt-curl-7843-9ad0b2/0.log"
Sep 08 04:26:17.908713 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:17.908685    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7843-9ad0b2"]
Sep 08 04:26:17.911307 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:17.911286    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7843-9ad0b2"]
Sep 08 04:26:17.987101 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:17.987048    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7frw\" (UniqueName: \"kubernetes.io/projected/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225-kube-api-access-g7frw\") pod \"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225\" (UID: \"8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225\") "
Sep 08 04:26:17.989058 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:17.989036    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225-kube-api-access-g7frw" (OuterVolumeSpecName: "kube-api-access-g7frw") pod "8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225" (UID: "8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225"). InnerVolumeSpecName "kube-api-access-g7frw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:18.029480 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.029458    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7843-567a70"]
Sep 08 04:26:18.029810 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.029798    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225" containerName="mt-curl-7843-9ad0b2"
Sep 08 04:26:18.029859 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.029812    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225" containerName="mt-curl-7843-9ad0b2"
Sep 08 04:26:18.029892 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.029870    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225" containerName="mt-curl-7843-9ad0b2"
Sep 08 04:26:18.032613 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.032599    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:18.042281 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.042259    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7843-567a70"]
Sep 08 04:26:18.088331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.088308    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7frw\" (UniqueName: \"kubernetes.io/projected/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225-kube-api-access-g7frw\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:18.160024 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.159998    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225" path="/var/lib/kubelet/pods/8bbf84c9-b6c3-41a7-bd5e-fc4e3a962225/volumes"
Sep 08 04:26:18.189642 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.189621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlvx\" (UniqueName: \"kubernetes.io/projected/ed193606-97c5-4503-9b87-fed81ab4cb83-kube-api-access-fzlvx\") pod \"mt-curl-7843-567a70\" (UID: \"ed193606-97c5-4503-9b87-fed81ab4cb83\") " pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:18.290835 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.290814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlvx\" (UniqueName: \"kubernetes.io/projected/ed193606-97c5-4503-9b87-fed81ab4cb83-kube-api-access-fzlvx\") pod \"mt-curl-7843-567a70\" (UID: \"ed193606-97c5-4503-9b87-fed81ab4cb83\") " pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:18.299571 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.299549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlvx\" (UniqueName: \"kubernetes.io/projected/ed193606-97c5-4503-9b87-fed81ab4cb83-kube-api-access-fzlvx\") pod \"mt-curl-7843-567a70\" (UID: \"ed193606-97c5-4503-9b87-fed81ab4cb83\") " pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:18.342572 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.342552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:18.609893 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.609865    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7843-567a70"]
Sep 08 04:26:18.612308 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:26:18.612269    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded193606_97c5_4503_9b87_fed81ab4cb83.slice/crio-6db14a6cceb54ff24afebd9054a6914acd599d46e20b48d2d6ca47e9f0f19fce WatchSource:0}: Error finding container 6db14a6cceb54ff24afebd9054a6914acd599d46e20b48d2d6ca47e9f0f19fce: Status 404 returned error can't find the container with id 6db14a6cceb54ff24afebd9054a6914acd599d46e20b48d2d6ca47e9f0f19fce
Sep 08 04:26:18.727640 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.727604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7843-567a70" event={"ID":"ed193606-97c5-4503-9b87-fed81ab4cb83","Type":"ContainerStarted","Data":"6db14a6cceb54ff24afebd9054a6914acd599d46e20b48d2d6ca47e9f0f19fce"}
Sep 08 04:26:18.728914 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.728895    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-9ad0b2"
Sep 08 04:26:18.729007 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:18.728897    2583 scope.go:117] "RemoveContainer" containerID="0ac440e91306d28cc5030ec01afc7a711c069943c5bc87a1377bb4999872fe44"
Sep 08 04:26:19.735772 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:19.735716    2583 generic.go:356] "Generic (PLEG): container finished" podID="ed193606-97c5-4503-9b87-fed81ab4cb83" containerID="b5275862840197ed0efb0bf570a0e148bcdf5879714aab19cfcd020d8e16ff6b" exitCode=0
Sep 08 04:26:19.736155 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:19.735800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7843-567a70" event={"ID":"ed193606-97c5-4503-9b87-fed81ab4cb83","Type":"ContainerDied","Data":"b5275862840197ed0efb0bf570a0e148bcdf5879714aab19cfcd020d8e16ff6b"}
Sep 08 04:26:20.322155 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.322126    2583 scope.go:117] "RemoveContainer" containerID="ff48df36b064699af47bbae64c0ab8952909b6c62cf2a224a6504b212c4d2316"
Sep 08 04:26:20.329968 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.329947    2583 scope.go:117] "RemoveContainer" containerID="490a2d3b1b0bb50c9845abc2c9db9804276adafba614411db499cfb47f5be262"
Sep 08 04:26:20.875193 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.875160    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:20.901414 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.901382    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7843-567a70_ed193606-97c5-4503-9b87-fed81ab4cb83/mt-curl-7843-567a70/0.log"
Sep 08 04:26:20.924770 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.924732    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7843-567a70"]
Sep 08 04:26:20.928293 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.928273    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7843-567a70"]
Sep 08 04:26:20.965606 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:20.965580    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:26:21.011935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:21.011915    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzlvx\" (UniqueName: \"kubernetes.io/projected/ed193606-97c5-4503-9b87-fed81ab4cb83-kube-api-access-fzlvx\") pod \"ed193606-97c5-4503-9b87-fed81ab4cb83\" (UID: \"ed193606-97c5-4503-9b87-fed81ab4cb83\") "
Sep 08 04:26:21.013844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:21.013818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed193606-97c5-4503-9b87-fed81ab4cb83-kube-api-access-fzlvx" (OuterVolumeSpecName: "kube-api-access-fzlvx") pod "ed193606-97c5-4503-9b87-fed81ab4cb83" (UID: "ed193606-97c5-4503-9b87-fed81ab4cb83"). InnerVolumeSpecName "kube-api-access-fzlvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:21.113143 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:21.113123    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlvx\" (UniqueName: \"kubernetes.io/projected/ed193606-97c5-4503-9b87-fed81ab4cb83-kube-api-access-fzlvx\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:21.746941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:21.746916    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db14a6cceb54ff24afebd9054a6914acd599d46e20b48d2d6ca47e9f0f19fce"
Sep 08 04:26:21.746941 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:21.746927    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:21.756459 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:21.756434    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7843-567a70\" is forbidden: User \"system:node:ip-10-0-130-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.ec2.internal' and this object" podUID="ed193606-97c5-4503-9b87-fed81ab4cb83" pod="opendatahub/mt-curl-7843-567a70"
Sep 08 04:26:22.159411 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:22.159384    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed193606-97c5-4503-9b87-fed81ab4cb83" path="/var/lib/kubelet/pods/ed193606-97c5-4503-9b87-fed81ab4cb83/volumes"
Sep 08 04:26:23.021482 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:23.021449    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"]
Sep 08 04:26:23.967462 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:23.967431    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"]
Sep 08 04:26:25.612700 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:25.612667    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"]
Sep 08 04:26:25.613098 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:25.612907    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" podUID="453a20f6-c45e-4ff9-ab98-7eb1add12062" containerName="maas-api" containerID="cri-o://3757f74f36c8252051a2b7791dc3878e96e2a87290633c294052f29763a542b7" gracePeriod=30
Sep 08 04:26:25.765485 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:25.765453    2583 generic.go:356] "Generic (PLEG): container finished" podID="453a20f6-c45e-4ff9-ab98-7eb1add12062" containerID="3757f74f36c8252051a2b7791dc3878e96e2a87290633c294052f29763a542b7" exitCode=0
Sep 08 04:26:25.765638 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:25.765530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" event={"ID":"453a20f6-c45e-4ff9-ab98-7eb1add12062","Type":"ContainerDied","Data":"3757f74f36c8252051a2b7791dc3878e96e2a87290633c294052f29763a542b7"}
Sep 08 04:26:25.862049 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:25.862023    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:26:26.044550 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.044517    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bb77\" (UniqueName: \"kubernetes.io/projected/453a20f6-c45e-4ff9-ab98-7eb1add12062-kube-api-access-8bb77\") pod \"453a20f6-c45e-4ff9-ab98-7eb1add12062\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") "
Sep 08 04:26:26.044712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.044589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453a20f6-c45e-4ff9-ab98-7eb1add12062-openshift-service-ca\") pod \"453a20f6-c45e-4ff9-ab98-7eb1add12062\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") "
Sep 08 04:26:26.044712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.044647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-metrics-certs\") pod \"453a20f6-c45e-4ff9-ab98-7eb1add12062\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") "
Sep 08 04:26:26.044712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.044677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls\") pod \"453a20f6-c45e-4ff9-ab98-7eb1add12062\" (UID: \"453a20f6-c45e-4ff9-ab98-7eb1add12062\") "
Sep 08 04:26:26.045026 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.045000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a20f6-c45e-4ff9-ab98-7eb1add12062-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "453a20f6-c45e-4ff9-ab98-7eb1add12062" (UID: "453a20f6-c45e-4ff9-ab98-7eb1add12062"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:26:26.046684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.046662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a20f6-c45e-4ff9-ab98-7eb1add12062-kube-api-access-8bb77" (OuterVolumeSpecName: "kube-api-access-8bb77") pod "453a20f6-c45e-4ff9-ab98-7eb1add12062" (UID: "453a20f6-c45e-4ff9-ab98-7eb1add12062"). InnerVolumeSpecName "kube-api-access-8bb77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:26.046794 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.046725    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "453a20f6-c45e-4ff9-ab98-7eb1add12062" (UID: "453a20f6-c45e-4ff9-ab98-7eb1add12062"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:26.046847 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.046831    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "453a20f6-c45e-4ff9-ab98-7eb1add12062" (UID: "453a20f6-c45e-4ff9-ab98-7eb1add12062"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:26.145982 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.145958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bb77\" (UniqueName: \"kubernetes.io/projected/453a20f6-c45e-4ff9-ab98-7eb1add12062-kube-api-access-8bb77\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:26.145982 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.145980    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/453a20f6-c45e-4ff9-ab98-7eb1add12062-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:26.146102 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.145990    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:26.146102 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.145999    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/453a20f6-c45e-4ff9-ab98-7eb1add12062-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:26.771500 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.771482    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"
Sep 08 04:26:26.771871 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.771482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp" event={"ID":"453a20f6-c45e-4ff9-ab98-7eb1add12062","Type":"ContainerDied","Data":"e18a2edba730d2da8417180dc43a662601a83a0c04775289b2419d677bcc0724"}
Sep 08 04:26:26.771871 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.771626    2583 scope.go:117] "RemoveContainer" containerID="3757f74f36c8252051a2b7791dc3878e96e2a87290633c294052f29763a542b7"
Sep 08 04:26:26.788246 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.788225    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"]
Sep 08 04:26:26.794720 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:26.794697    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-75264d-98959dbc6-2gchp"]
Sep 08 04:26:28.159648 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:28.159608    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453a20f6-c45e-4ff9-ab98-7eb1add12062" path="/var/lib/kubelet/pods/453a20f6-c45e-4ff9-ab98-7eb1add12062/volumes"
Sep 08 04:26:31.064135 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:31.064105    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:26:38.040314 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.040269    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj" podUID="3fbf9a60-f0b2-4a6f-9396-392982dd06e5" containerName="main" containerID="cri-o://c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73" gracePeriod=45
Sep 08 04:26:38.278196 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.278171    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:26:38.328051 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.327983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tmp-dir\") pod \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") "
Sep 08 04:26:38.328051 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-dshm\") pod \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") "
Sep 08 04:26:38.328236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-home\") pod \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") "
Sep 08 04:26:38.328236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hwth\" (UniqueName: \"kubernetes.io/projected/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-kube-api-access-9hwth\") pod \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") "
Sep 08 04:26:38.328236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328174    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tls-certs\") pod \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") "
Sep 08 04:26:38.328236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-model-cache\") pod \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\" (UID: \"3fbf9a60-f0b2-4a6f-9396-392982dd06e5\") "
Sep 08 04:26:38.328448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fbf9a60-f0b2-4a6f-9396-392982dd06e5" (UID: "3fbf9a60-f0b2-4a6f-9396-392982dd06e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:38.328448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-home" (OuterVolumeSpecName: "home") pod "3fbf9a60-f0b2-4a6f-9396-392982dd06e5" (UID: "3fbf9a60-f0b2-4a6f-9396-392982dd06e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:38.328557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328494    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:38.328557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328506    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:38.328557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.328502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fbf9a60-f0b2-4a6f-9396-392982dd06e5" (UID: "3fbf9a60-f0b2-4a6f-9396-392982dd06e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:38.330077 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.330051    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-dshm" (OuterVolumeSpecName: "dshm") pod "3fbf9a60-f0b2-4a6f-9396-392982dd06e5" (UID: "3fbf9a60-f0b2-4a6f-9396-392982dd06e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:38.330183 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.330162    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-kube-api-access-9hwth" (OuterVolumeSpecName: "kube-api-access-9hwth") pod "3fbf9a60-f0b2-4a6f-9396-392982dd06e5" (UID: "3fbf9a60-f0b2-4a6f-9396-392982dd06e5"). InnerVolumeSpecName "kube-api-access-9hwth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:38.330243 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.330220    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fbf9a60-f0b2-4a6f-9396-392982dd06e5" (UID: "3fbf9a60-f0b2-4a6f-9396-392982dd06e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:38.429240 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.429213    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hwth\" (UniqueName: \"kubernetes.io/projected/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-kube-api-access-9hwth\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:38.429240 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.429237    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:38.429365 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.429248    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:38.429365 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.429256    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fbf9a60-f0b2-4a6f-9396-392982dd06e5-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:38.830808 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.830777    2583 generic.go:356] "Generic (PLEG): container finished" podID="3fbf9a60-f0b2-4a6f-9396-392982dd06e5" containerID="c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73" exitCode=0
Sep 08 04:26:38.830918 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.830811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj" event={"ID":"3fbf9a60-f0b2-4a6f-9396-392982dd06e5","Type":"ContainerDied","Data":"c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73"}
Sep 08 04:26:38.830918 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.830815    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"
Sep 08 04:26:38.830918 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.830840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj" event={"ID":"3fbf9a60-f0b2-4a6f-9396-392982dd06e5","Type":"ContainerDied","Data":"802e3f9c3b3c53046237be6a0280584a98708d5bef75f87cdf37a8777dbb40de"}
Sep 08 04:26:38.830918 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.830857    2583 scope.go:117] "RemoveContainer" containerID="c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73"
Sep 08 04:26:38.839134 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.839114    2583 scope.go:117] "RemoveContainer" containerID="c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73"
Sep 08 04:26:38.839369 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:38.839350    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73\": container with ID starting with c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73 not found: ID does not exist" containerID="c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73"
Sep 08 04:26:38.839428 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.839374    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73"} err="failed to get container status \"c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73\": rpc error: code = NotFound desc = could not find container \"c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73\": container with ID starting with c99b5f4095433af0bd11fd8d603c9f4274e2ff7c7971f3363a879a47df5ece73 not found: ID does not exist"
Sep 08 04:26:38.852295 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.852274    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"]
Sep 08 04:26:38.857527 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.857508    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/sub-test-model-c2a8bb-kserve-7fbb56d4bc-wf4tj"]
Sep 08 04:26:38.986126 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:38.986099    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn" podUID="9641b082-71ba-4afe-a4f7-03e456f20ab5" containerName="main" containerID="cri-o://c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1" gracePeriod=45
Sep 08 04:26:39.235836 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.235811    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:26:39.335358 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335331    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-home\") pod \"9641b082-71ba-4afe-a4f7-03e456f20ab5\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") "
Sep 08 04:26:39.335518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-dshm\") pod \"9641b082-71ba-4afe-a4f7-03e456f20ab5\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") "
Sep 08 04:26:39.335518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335392    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zj27\" (UniqueName: \"kubernetes.io/projected/9641b082-71ba-4afe-a4f7-03e456f20ab5-kube-api-access-8zj27\") pod \"9641b082-71ba-4afe-a4f7-03e456f20ab5\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") "
Sep 08 04:26:39.335518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335420    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-model-cache\") pod \"9641b082-71ba-4afe-a4f7-03e456f20ab5\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") "
Sep 08 04:26:39.335518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335435    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-tmp-dir\") pod \"9641b082-71ba-4afe-a4f7-03e456f20ab5\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") "
Sep 08 04:26:39.335518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9641b082-71ba-4afe-a4f7-03e456f20ab5-tls-certs\") pod \"9641b082-71ba-4afe-a4f7-03e456f20ab5\" (UID: \"9641b082-71ba-4afe-a4f7-03e456f20ab5\") "
Sep 08 04:26:39.335850 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335608    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-home" (OuterVolumeSpecName: "home") pod "9641b082-71ba-4afe-a4f7-03e456f20ab5" (UID: "9641b082-71ba-4afe-a4f7-03e456f20ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:39.335850 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335712    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9641b082-71ba-4afe-a4f7-03e456f20ab5" (UID: "9641b082-71ba-4afe-a4f7-03e456f20ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:39.335850 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335729    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9641b082-71ba-4afe-a4f7-03e456f20ab5" (UID: "9641b082-71ba-4afe-a4f7-03e456f20ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:39.335850 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335794    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:39.335850 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.335807    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:39.337306 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.337279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641b082-71ba-4afe-a4f7-03e456f20ab5-kube-api-access-8zj27" (OuterVolumeSpecName: "kube-api-access-8zj27") pod "9641b082-71ba-4afe-a4f7-03e456f20ab5" (UID: "9641b082-71ba-4afe-a4f7-03e456f20ab5"). InnerVolumeSpecName "kube-api-access-8zj27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:39.337404 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.337390    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-dshm" (OuterVolumeSpecName: "dshm") pod "9641b082-71ba-4afe-a4f7-03e456f20ab5" (UID: "9641b082-71ba-4afe-a4f7-03e456f20ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:26:39.337443 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.337391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9641b082-71ba-4afe-a4f7-03e456f20ab5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9641b082-71ba-4afe-a4f7-03e456f20ab5" (UID: "9641b082-71ba-4afe-a4f7-03e456f20ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:39.436451 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.436399    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:39.436451 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.436419    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zj27\" (UniqueName: \"kubernetes.io/projected/9641b082-71ba-4afe-a4f7-03e456f20ab5-kube-api-access-8zj27\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:39.436451 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.436429    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9641b082-71ba-4afe-a4f7-03e456f20ab5-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:39.436451 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.436437    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9641b082-71ba-4afe-a4f7-03e456f20ab5-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:39.836998 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.836969    2583 generic.go:356] "Generic (PLEG): container finished" podID="9641b082-71ba-4afe-a4f7-03e456f20ab5" containerID="c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1" exitCode=0
Sep 08 04:26:39.837104 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.837016    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"
Sep 08 04:26:39.837104 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.837053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn" event={"ID":"9641b082-71ba-4afe-a4f7-03e456f20ab5","Type":"ContainerDied","Data":"c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1"}
Sep 08 04:26:39.837104 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.837090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn" event={"ID":"9641b082-71ba-4afe-a4f7-03e456f20ab5","Type":"ContainerDied","Data":"0b46b2f8af97f32abfb1631ea036d20c27e40a65ee645c032a6cf684f05cfffb"}
Sep 08 04:26:39.837217 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.837106    2583 scope.go:117] "RemoveContainer" containerID="c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1"
Sep 08 04:26:39.845553 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.845537    2583 scope.go:117] "RemoveContainer" containerID="c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1"
Sep 08 04:26:39.845844 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:39.845823    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1\": container with ID starting with c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1 not found: ID does not exist" containerID="c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1"
Sep 08 04:26:39.845905 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.845853    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1"} err="failed to get container status \"c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1\": rpc error: code = NotFound desc = could not find container \"c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1\": container with ID starting with c55579faafad5a1c9f29a9ddc2a27b2037c9d612f695956e4630736e3ae67bf1 not found: ID does not exist"
Sep 08 04:26:39.858014 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.857993    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"]
Sep 08 04:26:39.862057 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:39.862037    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/sub-test-model-720789-kserve-bb55b55b5-gs5gn"]
Sep 08 04:26:40.159299 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:40.159251    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbf9a60-f0b2-4a6f-9396-392982dd06e5" path="/var/lib/kubelet/pods/3fbf9a60-f0b2-4a6f-9396-392982dd06e5/volumes"
Sep 08 04:26:40.159608 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:40.159594    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9641b082-71ba-4afe-a4f7-03e456f20ab5" path="/var/lib/kubelet/pods/9641b082-71ba-4afe-a4f7-03e456f20ab5/volumes"
Sep 08 04:26:43.846196 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846166    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"]
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846464    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbf9a60-f0b2-4a6f-9396-392982dd06e5" containerName="main"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846476    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbf9a60-f0b2-4a6f-9396-392982dd06e5" containerName="main"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846483    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9641b082-71ba-4afe-a4f7-03e456f20ab5" containerName="main"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846488    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9641b082-71ba-4afe-a4f7-03e456f20ab5" containerName="main"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846496    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed193606-97c5-4503-9b87-fed81ab4cb83" containerName="mt-curl-7843-567a70"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846502    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed193606-97c5-4503-9b87-fed81ab4cb83" containerName="mt-curl-7843-567a70"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846512    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453a20f6-c45e-4ff9-ab98-7eb1add12062" containerName="maas-api"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846518    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="453a20f6-c45e-4ff9-ab98-7eb1add12062" containerName="maas-api"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846568    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed193606-97c5-4503-9b87-fed81ab4cb83" containerName="mt-curl-7843-567a70"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846576    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbf9a60-f0b2-4a6f-9396-392982dd06e5" containerName="main"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846585    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="453a20f6-c45e-4ff9-ab98-7eb1add12062" containerName="maas-api"
Sep 08 04:26:43.846635 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.846591    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9641b082-71ba-4afe-a4f7-03e456f20ab5" containerName="main"
Sep 08 04:26:43.850811 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.850794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.853237 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.853216    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"kube-root-ca.crt\""
Sep 08 04:26:43.854138 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.854099    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"default-dockercfg-w6tcb\""
Sep 08 04:26:43.854252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.854145    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"openshift-service-ca.crt\""
Sep 08 04:26:43.854252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.854167    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2a8bb\"/\"rate-test-model-c2a8bb-kserve-self-signed-certs\""
Sep 08 04:26:43.865578 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.863791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"]
Sep 08 04:26:43.867406 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.867373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-dshm\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.867520 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.867430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-home\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.867580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.867526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnvp\" (UniqueName: \"kubernetes.io/projected/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-kube-api-access-wnnvp\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.867674 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.867599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tmp-dir\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.867791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.867732    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tls-certs\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.868622 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.868601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-model-cache\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.969877 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.969853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-dshm\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.969977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.969880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-home\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.969977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.969902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnvp\" (UniqueName: \"kubernetes.io/projected/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-kube-api-access-wnnvp\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.970047 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.970015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tmp-dir\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.970083 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.970063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tls-certs\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.970127 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.970107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-model-cache\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.970247 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.970227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-home\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.970388 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.970366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tmp-dir\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.970458 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.970398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-model-cache\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.971989 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.971970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-dshm\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.972480 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.972458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tls-certs\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:43.977449 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:43.977429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnvp\" (UniqueName: \"kubernetes.io/projected/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-kube-api-access-wnnvp\") pod \"rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") " pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:44.162718 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.162667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:44.232586 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.232557    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"]
Sep 08 04:26:44.232821 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.232780    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69" podUID="279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" containerName="maas-api" containerID="cri-o://1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f" gracePeriod=30
Sep 08 04:26:44.295211 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.295176    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"]
Sep 08 04:26:44.295468 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.295416    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d" podUID="28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" containerName="payload-processing" containerID="cri-o://46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe" gracePeriod=30
Sep 08 04:26:44.302590 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.302561    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"]
Sep 08 04:26:44.302906 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.302839    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m" podUID="288b1513-588b-4753-aef3-09dee16321dc" containerName="payload-pre-processing" containerID="cri-o://37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51" gracePeriod=30
Sep 08 04:26:44.433553 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.433520    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"]
Sep 08 04:26:44.436080 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:26:44.436050    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod341cdb3e_8e8e_4c3e_bb99_f87730a6833a.slice/crio-30761a182dbe1a3f9655ee2ed6539f521196d4ada5b5ca257bc6fee7d0b63cff WatchSource:0}: Error finding container 30761a182dbe1a3f9655ee2ed6539f521196d4ada5b5ca257bc6fee7d0b63cff: Status 404 returned error can't find the container with id 30761a182dbe1a3f9655ee2ed6539f521196d4ada5b5ca257bc6fee7d0b63cff
Sep 08 04:26:44.546498 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.546469    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:26:44.575486 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.575463    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg9km\" (UniqueName: \"kubernetes.io/projected/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-kube-api-access-xg9km\") pod \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") "
Sep 08 04:26:44.575646 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.575498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-plugins-config-volume\") pod \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\" (UID: \"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a\") "
Sep 08 04:26:44.576063 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.576029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" (UID: "28057ce3-03cf-4f45-91a0-a9c4d6c1d68a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:26:44.578537 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.578510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-kube-api-access-xg9km" (OuterVolumeSpecName: "kube-api-access-xg9km") pod "28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" (UID: "28057ce3-03cf-4f45-91a0-a9c4d6c1d68a"). InnerVolumeSpecName "kube-api-access-xg9km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:44.589001 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.588980    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:26:44.598003 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.597982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:26:44.676432 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txpbz\" (UniqueName: \"kubernetes.io/projected/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-kube-api-access-txpbz\") pod \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") "
Sep 08 04:26:44.676432 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjxwq\" (UniqueName: \"kubernetes.io/projected/288b1513-588b-4753-aef3-09dee16321dc-kube-api-access-vjxwq\") pod \"288b1513-588b-4753-aef3-09dee16321dc\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") "
Sep 08 04:26:44.676622 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288b1513-588b-4753-aef3-09dee16321dc-plugins-config-volume\") pod \"288b1513-588b-4753-aef3-09dee16321dc\" (UID: \"288b1513-588b-4753-aef3-09dee16321dc\") "
Sep 08 04:26:44.676622 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-maas-api-tls\") pod \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") "
Sep 08 04:26:44.676622 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-metrics-certs\") pod \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") "
Sep 08 04:26:44.676622 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-openshift-service-ca\") pod \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\" (UID: \"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d\") "
Sep 08 04:26:44.676916 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676896    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.676981 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676922    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9km\" (UniqueName: \"kubernetes.io/projected/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a-kube-api-access-xg9km\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.676981 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.676924    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288b1513-588b-4753-aef3-09dee16321dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "288b1513-588b-4753-aef3-09dee16321dc" (UID: "288b1513-588b-4753-aef3-09dee16321dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:26:44.677082 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.677017    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" (UID: "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:26:44.678509 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.678481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-kube-api-access-txpbz" (OuterVolumeSpecName: "kube-api-access-txpbz") pod "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" (UID: "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d"). InnerVolumeSpecName "kube-api-access-txpbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:44.678595 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.678552    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" (UID: "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:44.678637 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.678613    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288b1513-588b-4753-aef3-09dee16321dc-kube-api-access-vjxwq" (OuterVolumeSpecName: "kube-api-access-vjxwq") pod "288b1513-588b-4753-aef3-09dee16321dc" (UID: "288b1513-588b-4753-aef3-09dee16321dc"). InnerVolumeSpecName "kube-api-access-vjxwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:26:44.678892 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.678873    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" (UID: "279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:26:44.777935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.777903    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288b1513-588b-4753-aef3-09dee16321dc-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.777935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.777933    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.777935 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.777943    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.778095 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.777951    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.778095 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.777960    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txpbz\" (UniqueName: \"kubernetes.io/projected/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d-kube-api-access-txpbz\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.778095 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.777968    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxwq\" (UniqueName: \"kubernetes.io/projected/288b1513-588b-4753-aef3-09dee16321dc-kube-api-access-vjxwq\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:26:44.865252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.865224    2583 generic.go:356] "Generic (PLEG): container finished" podID="288b1513-588b-4753-aef3-09dee16321dc" containerID="37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51" exitCode=0
Sep 08 04:26:44.865546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.865273    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"
Sep 08 04:26:44.865546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.865310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m" event={"ID":"288b1513-588b-4753-aef3-09dee16321dc","Type":"ContainerDied","Data":"37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51"}
Sep 08 04:26:44.865546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.865366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m" event={"ID":"288b1513-588b-4753-aef3-09dee16321dc","Type":"ContainerDied","Data":"22951c6c05835ee4ed1a1e144f04c87e7849287441d734526d58866991176486"}
Sep 08 04:26:44.865546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.865380    2583 scope.go:117] "RemoveContainer" containerID="37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51"
Sep 08 04:26:44.866701 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.866677    2583 generic.go:356] "Generic (PLEG): container finished" podID="28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" containerID="46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe" exitCode=0
Sep 08 04:26:44.866816 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.866715    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"
Sep 08 04:26:44.866816 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.866775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d" event={"ID":"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a","Type":"ContainerDied","Data":"46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe"}
Sep 08 04:26:44.866923 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.866835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d" event={"ID":"28057ce3-03cf-4f45-91a0-a9c4d6c1d68a","Type":"ContainerDied","Data":"bdfd4ea4e5fbe595fdbee3c31202e456f90768d9d3f6e17291f001303be2bbe6"}
Sep 08 04:26:44.868724 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.868691    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6" event={"ID":"341cdb3e-8e8e-4c3e-bb99-f87730a6833a","Type":"ContainerStarted","Data":"9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d"}
Sep 08 04:26:44.868894 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.868727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6" event={"ID":"341cdb3e-8e8e-4c3e-bb99-f87730a6833a","Type":"ContainerStarted","Data":"30761a182dbe1a3f9655ee2ed6539f521196d4ada5b5ca257bc6fee7d0b63cff"}
Sep 08 04:26:44.870193 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.870171    2583 generic.go:356] "Generic (PLEG): container finished" podID="279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" containerID="1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f" exitCode=0
Sep 08 04:26:44.870271 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.870221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69" event={"ID":"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d","Type":"ContainerDied","Data":"1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f"}
Sep 08 04:26:44.870271 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.870259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69" event={"ID":"279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d","Type":"ContainerDied","Data":"27e373d48eb22c5eba0bed1ab36ec4740648e449006ac3cc57588348c16e0351"}
Sep 08 04:26:44.870364 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.870222    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"
Sep 08 04:26:44.874879 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.874849    2583 scope.go:117] "RemoveContainer" containerID="37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51"
Sep 08 04:26:44.875198 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:44.875174    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51\": container with ID starting with 37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51 not found: ID does not exist" containerID="37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51"
Sep 08 04:26:44.875275 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.875209    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51"} err="failed to get container status \"37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51\": rpc error: code = NotFound desc = could not find container \"37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51\": container with ID starting with 37ffde3bd741f69667d06a180f0f4e9a2dfd6072f2336e9b52078e707feb7c51 not found: ID does not exist"
Sep 08 04:26:44.875275 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.875232    2583 scope.go:117] "RemoveContainer" containerID="46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe"
Sep 08 04:26:44.883546 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.883529    2583 scope.go:117] "RemoveContainer" containerID="46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe"
Sep 08 04:26:44.883834 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:44.883816    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe\": container with ID starting with 46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe not found: ID does not exist" containerID="46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe"
Sep 08 04:26:44.883896 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.883840    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe"} err="failed to get container status \"46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe\": rpc error: code = NotFound desc = could not find container \"46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe\": container with ID starting with 46b5b9839a07576249a3b3ec122a27ff89933147657a92e94818b46367bbf9fe not found: ID does not exist"
Sep 08 04:26:44.883896 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.883855    2583 scope.go:117] "RemoveContainer" containerID="1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f"
Sep 08 04:26:44.891018 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.890990    2583 scope.go:117] "RemoveContainer" containerID="1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f"
Sep 08 04:26:44.891273 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:26:44.891255    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f\": container with ID starting with 1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f not found: ID does not exist" containerID="1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f"
Sep 08 04:26:44.891331 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.891280    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f"} err="failed to get container status \"1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f\": rpc error: code = NotFound desc = could not find container \"1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f\": container with ID starting with 1e4f7c6e6f4eb3c9cfc6a55c8b725a4e09f8e509e422660b318a70e927f65d8f not found: ID does not exist"
Sep 08 04:26:44.895672 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.895636    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6" podStartSLOduration=1.895624124 podStartE2EDuration="1.895624124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:26:44.89388509 +0000 UTC m=+1585.252364651" watchObservedRunningTime="2026-09-08 04:26:44.895624124 +0000 UTC m=+1585.254103725"
Sep 08 04:26:44.911952 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.911933    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"]
Sep 08 04:26:44.916387 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.916368    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c2eafd-5f7655f74b-ljj2m"]
Sep 08 04:26:44.927765 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.927702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"]
Sep 08 04:26:44.931518 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.931498    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c2eafd-77ccb798b4-bnl69"]
Sep 08 04:26:44.942698 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.942676    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"]
Sep 08 04:26:44.946250 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:44.946232    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c2eafd-cb94c98d9-55w6d"]
Sep 08 04:26:46.159754 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:46.159707    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" path="/var/lib/kubelet/pods/279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d/volumes"
Sep 08 04:26:46.160088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:46.160052    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" path="/var/lib/kubelet/pods/28057ce3-03cf-4f45-91a0-a9c4d6c1d68a/volumes"
Sep 08 04:26:46.160352 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:46.160339    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="288b1513-588b-4753-aef3-09dee16321dc" path="/var/lib/kubelet/pods/288b1513-588b-4753-aef3-09dee16321dc/volumes"
Sep 08 04:26:47.457611 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:47.457581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:26:54.162772 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:54.162723    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:54.163162 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:54.162782    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:54.174756 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:54.174724    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:26:54.932900 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:26:54.932870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:27:10.441158 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441120    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"]
Sep 08 04:27:10.441629 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441583    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288b1513-588b-4753-aef3-09dee16321dc" containerName="payload-pre-processing"
Sep 08 04:27:10.441629 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441601    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="288b1513-588b-4753-aef3-09dee16321dc" containerName="payload-pre-processing"
Sep 08 04:27:10.441703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441632    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" containerName="maas-api"
Sep 08 04:27:10.441703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441641    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" containerName="maas-api"
Sep 08 04:27:10.441703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441655    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" containerName="payload-processing"
Sep 08 04:27:10.441703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441664    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" containerName="payload-processing"
Sep 08 04:27:10.441854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441734    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="288b1513-588b-4753-aef3-09dee16321dc" containerName="payload-pre-processing"
Sep 08 04:27:10.441854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441760    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="28057ce3-03cf-4f45-91a0-a9c4d6c1d68a" containerName="payload-processing"
Sep 08 04:27:10.441854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.441768    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="279ffc93-94f5-4e85-b0f3-2e4db4b7bb6d" containerName="maas-api"
Sep 08 04:27:10.444729 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.444706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.448148 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.448111    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"kube-root-ca.crt\""
Sep 08 04:27:10.448148 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.448129    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"default-dockercfg-w5qg5\""
Sep 08 04:27:10.448325 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.448155    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"rate-test-model-720789-kserve-self-signed-certs\""
Sep 08 04:27:10.448325 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.448199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-720789\"/\"openshift-service-ca.crt\""
Sep 08 04:27:10.455913 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.455890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"]
Sep 08 04:27:10.551095 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.551067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-home\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.551248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.551154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tmp-dir\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.551248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.551184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-model-cache\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.551248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.551216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-dshm\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.551248 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.551238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tls-certs\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.551424 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.551265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7sq\" (UniqueName: \"kubernetes.io/projected/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-kube-api-access-jh7sq\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652357 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-home\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652493 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tmp-dir\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652493 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-model-cache\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652493 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-dshm\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652660 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tls-certs\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652660 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7sq\" (UniqueName: \"kubernetes.io/projected/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-kube-api-access-jh7sq\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652830 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-home\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tmp-dir\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.652886 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.652842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-model-cache\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.654698 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.654675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-dshm\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.654969 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.654952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tls-certs\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.662017 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.661995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7sq\" (UniqueName: \"kubernetes.io/projected/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-kube-api-access-jh7sq\") pod \"rate-test-model-720789-kserve-67b7b454dc-bztvk\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") " pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:10.753749 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:10.753725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:11.024172 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:11.024141    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"]
Sep 08 04:27:11.026547 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:27:11.026521    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fca6bce_f754_4a2d_a594_88d6d08fc6b9.slice/crio-f13271a78f63aae1be7226402f86ec9c8bb4b45fb0b754c5dc112eb21daed982 WatchSource:0}: Error finding container f13271a78f63aae1be7226402f86ec9c8bb4b45fb0b754c5dc112eb21daed982: Status 404 returned error can't find the container with id f13271a78f63aae1be7226402f86ec9c8bb4b45fb0b754c5dc112eb21daed982
Sep 08 04:27:11.997871 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:11.997840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk" event={"ID":"2fca6bce-f754-4a2d-a594-88d6d08fc6b9","Type":"ContainerStarted","Data":"a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f"}
Sep 08 04:27:11.998315 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:11.997878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk" event={"ID":"2fca6bce-f754-4a2d-a594-88d6d08fc6b9","Type":"ContainerStarted","Data":"f13271a78f63aae1be7226402f86ec9c8bb4b45fb0b754c5dc112eb21daed982"}
Sep 08 04:27:12.017407 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:12.017368    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk" podStartSLOduration=2.01735204 podStartE2EDuration="2.01735204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:27:12.016191463 +0000 UTC m=+1612.374671023" watchObservedRunningTime="2026-09-08 04:27:12.01735204 +0000 UTC m=+1612.375831601"
Sep 08 04:27:14.159659 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:14.159579    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:27:20.364267 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:20.364237    2583 scope.go:117] "RemoveContainer" containerID="c3b6faebfa6d767fc7a69539e3b58105123cb553c91fe23bb100480cf7eef827"
Sep 08 04:27:20.371641 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:20.371617    2583 scope.go:117] "RemoveContainer" containerID="8d647d7306f69ac21e0f5eb0ef318a80432f5c76c68e0ea28aaadcf9f306b821"
Sep 08 04:27:20.754786 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:20.754763    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:20.754879 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:20.754797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:20.767040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:20.767014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:21.051088 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:21.051034    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:27:25.747602 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:25.747572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:27:41.052174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:41.052121    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:27:47.456603 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:47.456568    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"]
Sep 08 04:27:48.367252 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:48.367218    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"]
Sep 08 04:27:52.150891 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:52.150854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:27:53.660058 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.660027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"]
Sep 08 04:27:53.663610 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.663588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.666112 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.666089    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:27:53.666227 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.666149    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 04:27:53.667066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.667041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 04:27:53.667185 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.667129    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npqmj\""
Sep 08 04:27:53.667185 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.667156    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-520856-serving-cert\""
Sep 08 04:27:53.674997 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.674973    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"]
Sep 08 04:27:53.759309 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.759279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb28\" (UniqueName: \"kubernetes.io/projected/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-kube-api-access-lsb28\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.759413 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.759325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.759467 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.759406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.759513 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.759484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-metrics-certs\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.860577 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.860552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb28\" (UniqueName: \"kubernetes.io/projected/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-kube-api-access-lsb28\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.860675 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.860586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.860675 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.860618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.860847 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:27:53.860695    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-serving-cert: secret "maas-api-e2e-ipp-a-520856-serving-cert" not found
Sep 08 04:27:53.860847 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.860732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-metrics-certs\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.860847 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:27:53.860779    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls podName:e6374bc2-ef3a-45f2-8e53-9ddd146215d1 nodeName:}" failed. No retries permitted until 2026-09-08 04:27:54.360759186 +0000 UTC m=+1654.719238724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls") pod "maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" (UID: "e6374bc2-ef3a-45f2-8e53-9ddd146215d1") : secret "maas-api-e2e-ipp-a-520856-serving-cert" not found
Sep 08 04:27:53.861226 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.861208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.863118 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.863100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-metrics-certs\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:53.869220 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:53.869195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb28\" (UniqueName: \"kubernetes.io/projected/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-kube-api-access-lsb28\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:54.363091 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.363058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:54.365690 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.365666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:54.367526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.367504    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"]
Sep 08 04:27:54.371227 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.371210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.373656 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.373627    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-520856\""
Sep 08 04:27:54.373800 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.373781    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-520856-dockercfg-dpzb7\""
Sep 08 04:27:54.378599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.378569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"]
Sep 08 04:27:54.464435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.464402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pc9\" (UniqueName: \"kubernetes.io/projected/7673b720-b142-4b96-bc39-d4389f598dd7-kube-api-access-95pc9\") pod \"payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.464552 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.464496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673b720-b142-4b96-bc39-d4389f598dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.565070 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.565039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673b720-b142-4b96-bc39-d4389f598dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.565182 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.565092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pc9\" (UniqueName: \"kubernetes.io/projected/7673b720-b142-4b96-bc39-d4389f598dd7-kube-api-access-95pc9\") pod \"payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.565808 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.565790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673b720-b142-4b96-bc39-d4389f598dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.574327 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.574304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pc9\" (UniqueName: \"kubernetes.io/projected/7673b720-b142-4b96-bc39-d4389f598dd7-kube-api-access-95pc9\") pod \"payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.577096 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.577077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:54.683883 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.683853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:54.848116 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.848074    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"]
Sep 08 04:27:54.850397 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:27:54.850375    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6374bc2_ef3a_45f2_8e53_9ddd146215d1.slice/crio-436c785869c9945641cd911a67756a478c4ec1b2e37e825e5bd3462a20277954 WatchSource:0}: Error finding container 436c785869c9945641cd911a67756a478c4ec1b2e37e825e5bd3462a20277954: Status 404 returned error can't find the container with id 436c785869c9945641cd911a67756a478c4ec1b2e37e825e5bd3462a20277954
Sep 08 04:27:54.954204 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:54.954144    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"]
Sep 08 04:27:54.956613 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:27:54.956581    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7673b720_b142_4b96_bc39_d4389f598dd7.slice/crio-b49094173dc6a4a34009718a6fe23e226b830b27871c6abc9fe87c7c79b33253 WatchSource:0}: Error finding container b49094173dc6a4a34009718a6fe23e226b830b27871c6abc9fe87c7c79b33253: Status 404 returned error can't find the container with id b49094173dc6a4a34009718a6fe23e226b830b27871c6abc9fe87c7c79b33253
Sep 08 04:27:55.189245 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:55.189208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v" event={"ID":"7673b720-b142-4b96-bc39-d4389f598dd7","Type":"ContainerStarted","Data":"d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410"}
Sep 08 04:27:55.189410 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:55.189289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v" event={"ID":"7673b720-b142-4b96-bc39-d4389f598dd7","Type":"ContainerStarted","Data":"b49094173dc6a4a34009718a6fe23e226b830b27871c6abc9fe87c7c79b33253"}
Sep 08 04:27:55.189410 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:55.189320    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:27:55.190443 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:55.190415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" event={"ID":"e6374bc2-ef3a-45f2-8e53-9ddd146215d1","Type":"ContainerStarted","Data":"436c785869c9945641cd911a67756a478c4ec1b2e37e825e5bd3462a20277954"}
Sep 08 04:27:55.211910 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:55.211824    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v" podStartSLOduration=1.211811375 podStartE2EDuration="1.211811375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:27:55.209515703 +0000 UTC m=+1655.567995262" watchObservedRunningTime="2026-09-08 04:27:55.211811375 +0000 UTC m=+1655.570290934"
Sep 08 04:27:56.198502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:56.198467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" event={"ID":"e6374bc2-ef3a-45f2-8e53-9ddd146215d1","Type":"ContainerStarted","Data":"41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed"}
Sep 08 04:27:56.198887 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:56.198539    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:27:56.218580 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:27:56.218535    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" podStartSLOduration=2.742799413 podStartE2EDuration="3.218522856s" totalImagesPullingTime="475.723443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:27:56.217061273 +0000 UTC m=+1656.575540833" watchObservedRunningTime="2026-09-08 04:27:56.218522856 +0000 UTC m=+1656.577002416"
Sep 08 04:28:02.208419 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.208380    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:28:02.475276 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.475172    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6" podUID="341cdb3e-8e8e-4c3e-bb99-f87730a6833a" containerName="main" containerID="cri-o://9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d" gracePeriod=45
Sep 08 04:28:02.716950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.716926    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:28:02.827499 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnnvp\" (UniqueName: \"kubernetes.io/projected/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-kube-api-access-wnnvp\") pod \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") "
Sep 08 04:28:02.827665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tls-certs\") pod \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") "
Sep 08 04:28:02.827665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tmp-dir\") pod \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") "
Sep 08 04:28:02.827665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-dshm\") pod \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") "
Sep 08 04:28:02.827665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827617    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-model-cache\") pod \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") "
Sep 08 04:28:02.827915 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827668    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-home\") pod \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\" (UID: \"341cdb3e-8e8e-4c3e-bb99-f87730a6833a\") "
Sep 08 04:28:02.827984 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "341cdb3e-8e8e-4c3e-bb99-f87730a6833a" (UID: "341cdb3e-8e8e-4c3e-bb99-f87730a6833a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:02.828037 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-model-cache" (OuterVolumeSpecName: "model-cache") pod "341cdb3e-8e8e-4c3e-bb99-f87730a6833a" (UID: "341cdb3e-8e8e-4c3e-bb99-f87730a6833a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:02.828037 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.827971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-home" (OuterVolumeSpecName: "home") pod "341cdb3e-8e8e-4c3e-bb99-f87730a6833a" (UID: "341cdb3e-8e8e-4c3e-bb99-f87730a6833a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:02.829632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.829606    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-dshm" (OuterVolumeSpecName: "dshm") pod "341cdb3e-8e8e-4c3e-bb99-f87730a6833a" (UID: "341cdb3e-8e8e-4c3e-bb99-f87730a6833a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:02.829733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.829662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "341cdb3e-8e8e-4c3e-bb99-f87730a6833a" (UID: "341cdb3e-8e8e-4c3e-bb99-f87730a6833a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:28:02.829733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.829676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-kube-api-access-wnnvp" (OuterVolumeSpecName: "kube-api-access-wnnvp") pod "341cdb3e-8e8e-4c3e-bb99-f87730a6833a" (UID: "341cdb3e-8e8e-4c3e-bb99-f87730a6833a"). InnerVolumeSpecName "kube-api-access-wnnvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:28:02.928712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.928686    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:02.928712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.928709    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:02.928878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.928722    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:02.928878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.928733    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:02.928878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.928767    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnnvp\" (UniqueName: \"kubernetes.io/projected/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-kube-api-access-wnnvp\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:02.928878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.928779    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341cdb3e-8e8e-4c3e-bb99-f87730a6833a-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:02.991066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:02.991043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:28:03.229775 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.229699    2583 generic.go:356] "Generic (PLEG): container finished" podID="341cdb3e-8e8e-4c3e-bb99-f87730a6833a" containerID="9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d" exitCode=0
Sep 08 04:28:03.229775 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.229767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"
Sep 08 04:28:03.230232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.229776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6" event={"ID":"341cdb3e-8e8e-4c3e-bb99-f87730a6833a","Type":"ContainerDied","Data":"9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d"}
Sep 08 04:28:03.230232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.229814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6" event={"ID":"341cdb3e-8e8e-4c3e-bb99-f87730a6833a","Type":"ContainerDied","Data":"30761a182dbe1a3f9655ee2ed6539f521196d4ada5b5ca257bc6fee7d0b63cff"}
Sep 08 04:28:03.230232 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.229832    2583 scope.go:117] "RemoveContainer" containerID="9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d"
Sep 08 04:28:03.237934 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.237915    2583 scope.go:117] "RemoveContainer" containerID="9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d"
Sep 08 04:28:03.238207 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:28:03.238191    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d\": container with ID starting with 9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d not found: ID does not exist" containerID="9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d"
Sep 08 04:28:03.238245 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.238217    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d"} err="failed to get container status \"9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d\": rpc error: code = NotFound desc = could not find container \"9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d\": container with ID starting with 9aeffe85eb00a5ec8d2f4be3488a9aa4c54adcb3e828910ec5b8e509af02c31d not found: ID does not exist"
Sep 08 04:28:03.250354 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.250331    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"]
Sep 08 04:28:03.252119 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.252099    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2a8bb/rate-test-model-c2a8bb-kserve-74bfd4778c-xd9n6"]
Sep 08 04:28:03.385959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.385930    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk" podUID="2fca6bce-f754-4a2d-a594-88d6d08fc6b9" containerName="main" containerID="cri-o://a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f" gracePeriod=45
Sep 08 04:28:03.624605 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.624583    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:28:03.734206 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tmp-dir\") pod \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") "
Sep 08 04:28:03.734346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734221    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh7sq\" (UniqueName: \"kubernetes.io/projected/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-kube-api-access-jh7sq\") pod \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") "
Sep 08 04:28:03.734346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-dshm\") pod \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") "
Sep 08 04:28:03.734346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-model-cache\") pod \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") "
Sep 08 04:28:03.734503 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tls-certs\") pod \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") "
Sep 08 04:28:03.734503 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-home\") pod \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\" (UID: \"2fca6bce-f754-4a2d-a594-88d6d08fc6b9\") "
Sep 08 04:28:03.734503 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734436    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fca6bce-f754-4a2d-a594-88d6d08fc6b9" (UID: "2fca6bce-f754-4a2d-a594-88d6d08fc6b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:03.734642 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fca6bce-f754-4a2d-a594-88d6d08fc6b9" (UID: "2fca6bce-f754-4a2d-a594-88d6d08fc6b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:03.734728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734709    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:03.734815 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734718    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-home" (OuterVolumeSpecName: "home") pod "2fca6bce-f754-4a2d-a594-88d6d08fc6b9" (UID: "2fca6bce-f754-4a2d-a594-88d6d08fc6b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:03.734815 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.734760    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:03.736253 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.736199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-kube-api-access-jh7sq" (OuterVolumeSpecName: "kube-api-access-jh7sq") pod "2fca6bce-f754-4a2d-a594-88d6d08fc6b9" (UID: "2fca6bce-f754-4a2d-a594-88d6d08fc6b9"). InnerVolumeSpecName "kube-api-access-jh7sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:28:03.736357 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.736330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fca6bce-f754-4a2d-a594-88d6d08fc6b9" (UID: "2fca6bce-f754-4a2d-a594-88d6d08fc6b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:28:03.736441 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.736420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-dshm" (OuterVolumeSpecName: "dshm") pod "2fca6bce-f754-4a2d-a594-88d6d08fc6b9" (UID: "2fca6bce-f754-4a2d-a594-88d6d08fc6b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:28:03.835386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.835364    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:03.835386 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.835383    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:03.835512 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.835392    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:03.835512 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:03.835401    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7sq\" (UniqueName: \"kubernetes.io/projected/2fca6bce-f754-4a2d-a594-88d6d08fc6b9-kube-api-access-jh7sq\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:28:04.159514 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.159488    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341cdb3e-8e8e-4c3e-bb99-f87730a6833a" path="/var/lib/kubelet/pods/341cdb3e-8e8e-4c3e-bb99-f87730a6833a/volumes"
Sep 08 04:28:04.236982 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.236953    2583 generic.go:356] "Generic (PLEG): container finished" podID="2fca6bce-f754-4a2d-a594-88d6d08fc6b9" containerID="a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f" exitCode=0
Sep 08 04:28:04.237295 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.236992    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"
Sep 08 04:28:04.237295 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.237021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk" event={"ID":"2fca6bce-f754-4a2d-a594-88d6d08fc6b9","Type":"ContainerDied","Data":"a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f"}
Sep 08 04:28:04.237295 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.237047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk" event={"ID":"2fca6bce-f754-4a2d-a594-88d6d08fc6b9","Type":"ContainerDied","Data":"f13271a78f63aae1be7226402f86ec9c8bb4b45fb0b754c5dc112eb21daed982"}
Sep 08 04:28:04.237295 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.237063    2583 scope.go:117] "RemoveContainer" containerID="a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f"
Sep 08 04:28:04.245136 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.245117    2583 scope.go:117] "RemoveContainer" containerID="a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f"
Sep 08 04:28:04.245401 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:28:04.245381    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f\": container with ID starting with a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f not found: ID does not exist" containerID="a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f"
Sep 08 04:28:04.245449 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.245412    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f"} err="failed to get container status \"a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f\": rpc error: code = NotFound desc = could not find container \"a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f\": container with ID starting with a2aff33da072dc2350cb9d9690e2f19a16ce958a3b36bc954c2a3ca901f7753f not found: ID does not exist"
Sep 08 04:28:04.255373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.255349    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"]
Sep 08 04:28:04.260475 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:04.260457    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-720789/rate-test-model-720789-kserve-67b7b454dc-bztvk"]
Sep 08 04:28:06.159357 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:06.159327    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fca6bce-f754-4a2d-a594-88d6d08fc6b9" path="/var/lib/kubelet/pods/2fca6bce-f754-4a2d-a594-88d6d08fc6b9/volumes"
Sep 08 04:28:06.200904 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:06.200879    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:28:22.779442 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779408    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"]
Sep 08 04:28:22.779834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779727    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fca6bce-f754-4a2d-a594-88d6d08fc6b9" containerName="main"
Sep 08 04:28:22.779834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779759    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fca6bce-f754-4a2d-a594-88d6d08fc6b9" containerName="main"
Sep 08 04:28:22.779834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779779    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341cdb3e-8e8e-4c3e-bb99-f87730a6833a" containerName="main"
Sep 08 04:28:22.779834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779785    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="341cdb3e-8e8e-4c3e-bb99-f87730a6833a" containerName="main"
Sep 08 04:28:22.779834 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779834    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fca6bce-f754-4a2d-a594-88d6d08fc6b9" containerName="main"
Sep 08 04:28:22.780005 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.779843    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="341cdb3e-8e8e-4c3e-bb99-f87730a6833a" containerName="main"
Sep 08 04:28:22.782925 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.782902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.785416 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.785392    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6c84ae-serving-cert\""
Sep 08 04:28:22.793891 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.793868    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"]
Sep 08 04:28:22.872708 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.872680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-metrics-certs\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.872833 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.872711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.872902 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.872839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b524-ca36-4095-896b-181ed32293af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.872902 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.872896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9pw\" (UniqueName: \"kubernetes.io/projected/bf90b524-ca36-4095-896b-181ed32293af-kube-api-access-td9pw\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.973512 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.973483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b524-ca36-4095-896b-181ed32293af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.973618 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.973533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9pw\" (UniqueName: \"kubernetes.io/projected/bf90b524-ca36-4095-896b-181ed32293af-kube-api-access-td9pw\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.973618 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.973583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-metrics-certs\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.973618 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.973610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.973828 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:28:22.973777    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-serving-cert: secret "maas-api-e2e-ipp-b-6c84ae-serving-cert" not found
Sep 08 04:28:22.973884 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:28:22.973852    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls podName:bf90b524-ca36-4095-896b-181ed32293af nodeName:}" failed. No retries permitted until 2026-09-08 04:28:23.473832046 +0000 UTC m=+1683.832311585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls") pod "maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" (UID: "bf90b524-ca36-4095-896b-181ed32293af") : secret "maas-api-e2e-ipp-b-6c84ae-serving-cert" not found
Sep 08 04:28:22.974223 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.974204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b524-ca36-4095-896b-181ed32293af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.975922 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.975899    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-metrics-certs\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:22.987649 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:22.987631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9pw\" (UniqueName: \"kubernetes.io/projected/bf90b524-ca36-4095-896b-181ed32293af-kube-api-access-td9pw\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:23.478215 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.478185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:23.480445 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.480409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:23.681432 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.681398    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"]
Sep 08 04:28:23.684962 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.684943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.687356 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.687334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6c84ae-dockercfg-hhl98\""
Sep 08 04:28:23.687633 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.687617    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6c84ae\""
Sep 08 04:28:23.694250 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.694226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:23.703565 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.703545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"]
Sep 08 04:28:23.780458 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.780424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpsk\" (UniqueName: \"kubernetes.io/projected/eff90492-5078-4ad1-b931-3accc3ad2c15-kube-api-access-2mpsk\") pod \"payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.780782 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.780484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff90492-5078-4ad1-b931-3accc3ad2c15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.804669 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.804645    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"]
Sep 08 04:28:23.808852 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.808836    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.820543 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.820521    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"]
Sep 08 04:28:23.881251 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.881223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpsk\" (UniqueName: \"kubernetes.io/projected/eff90492-5078-4ad1-b931-3accc3ad2c15-kube-api-access-2mpsk\") pod \"payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.881356 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.881264    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctldh\" (UniqueName: \"kubernetes.io/projected/a5e595b1-dd15-4819-9294-dc77c19eb2df-kube-api-access-ctldh\") pod \"payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.881356 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.881302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e595b1-dd15-4819-9294-dc77c19eb2df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.881356 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.881335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff90492-5078-4ad1-b931-3accc3ad2c15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.882009 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.881987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff90492-5078-4ad1-b931-3accc3ad2c15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.890958 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.890941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpsk\" (UniqueName: \"kubernetes.io/projected/eff90492-5078-4ad1-b931-3accc3ad2c15-kube-api-access-2mpsk\") pod \"payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:23.966576 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.966556    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"]
Sep 08 04:28:23.968590 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:28:23.968565    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf90b524_ca36_4095_896b_181ed32293af.slice/crio-938846d6e74b71807215813a8405340b2fbeee3b17238c664faa4b60523f28c0 WatchSource:0}: Error finding container 938846d6e74b71807215813a8405340b2fbeee3b17238c664faa4b60523f28c0: Status 404 returned error can't find the container with id 938846d6e74b71807215813a8405340b2fbeee3b17238c664faa4b60523f28c0
Sep 08 04:28:23.981954 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.981931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctldh\" (UniqueName: \"kubernetes.io/projected/a5e595b1-dd15-4819-9294-dc77c19eb2df-kube-api-access-ctldh\") pod \"payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.982057 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.981967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e595b1-dd15-4819-9294-dc77c19eb2df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.982824 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.982801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e595b1-dd15-4819-9294-dc77c19eb2df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.991677 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.991634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctldh\" (UniqueName: \"kubernetes.io/projected/a5e595b1-dd15-4819-9294-dc77c19eb2df-kube-api-access-ctldh\") pod \"payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:23.994500 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:23.994483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:24.119604 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:24.119573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:24.271524 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:24.271498    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"]
Sep 08 04:28:24.273507 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:28:24.273482    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff90492_5078_4ad1_b931_3accc3ad2c15.slice/crio-c80b16bef6930ae3b619f782edec058af096ef83ee4222690f672ec2ac195656 WatchSource:0}: Error finding container c80b16bef6930ae3b619f782edec058af096ef83ee4222690f672ec2ac195656: Status 404 returned error can't find the container with id c80b16bef6930ae3b619f782edec058af096ef83ee4222690f672ec2ac195656
Sep 08 04:28:24.323028 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:24.322998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv" event={"ID":"eff90492-5078-4ad1-b931-3accc3ad2c15","Type":"ContainerStarted","Data":"c80b16bef6930ae3b619f782edec058af096ef83ee4222690f672ec2ac195656"}
Sep 08 04:28:24.324168 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:24.324139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" event={"ID":"bf90b524-ca36-4095-896b-181ed32293af","Type":"ContainerStarted","Data":"938846d6e74b71807215813a8405340b2fbeee3b17238c664faa4b60523f28c0"}
Sep 08 04:28:24.391659 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:24.391636    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"]
Sep 08 04:28:24.393889 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:28:24.393862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e595b1_dd15_4819_9294_dc77c19eb2df.slice/crio-e10e420223de5a4820933a241747a36e649a7abb57cd5ea8cfa8478d29da22e6 WatchSource:0}: Error finding container e10e420223de5a4820933a241747a36e649a7abb57cd5ea8cfa8478d29da22e6: Status 404 returned error can't find the container with id e10e420223de5a4820933a241747a36e649a7abb57cd5ea8cfa8478d29da22e6
Sep 08 04:28:25.331436 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.331398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz" event={"ID":"a5e595b1-dd15-4819-9294-dc77c19eb2df","Type":"ContainerStarted","Data":"c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417"}
Sep 08 04:28:25.331894 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.331602    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:25.331894 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.331648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz" event={"ID":"a5e595b1-dd15-4819-9294-dc77c19eb2df","Type":"ContainerStarted","Data":"e10e420223de5a4820933a241747a36e649a7abb57cd5ea8cfa8478d29da22e6"}
Sep 08 04:28:25.333392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.333362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv" event={"ID":"eff90492-5078-4ad1-b931-3accc3ad2c15","Type":"ContainerStarted","Data":"bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c"}
Sep 08 04:28:25.333554 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.333511    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:25.335039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.335016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" event={"ID":"bf90b524-ca36-4095-896b-181ed32293af","Type":"ContainerStarted","Data":"d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75"}
Sep 08 04:28:25.335192 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.335170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:25.352021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.351981    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz" podStartSLOduration=2.351968126 podStartE2EDuration="2.351968126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:28:25.350201901 +0000 UTC m=+1685.708681463" watchObservedRunningTime="2026-09-08 04:28:25.351968126 +0000 UTC m=+1685.710447686"
Sep 08 04:28:25.368724 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.368681    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" podStartSLOduration=3.008307513 podStartE2EDuration="3.368668873s" totalImagesPullingTime="360.36136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:28:25.367025976 +0000 UTC m=+1685.725505549" watchObservedRunningTime="2026-09-08 04:28:25.368668873 +0000 UTC m=+1685.727148427"
Sep 08 04:28:25.391704 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:25.391662    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv" podStartSLOduration=2.391650019 podStartE2EDuration="2.391650019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:28:25.390202569 +0000 UTC m=+1685.748682132" watchObservedRunningTime="2026-09-08 04:28:25.391650019 +0000 UTC m=+1685.750129592"
Sep 08 04:28:28.153272 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:28.153240    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:28:31.345646 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:31.345611    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:28:36.342180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:36.342150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:28:36.342627 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:36.342307    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:28:51.984079 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:28:51.984000    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz_a5e595b1-dd15-4819-9294-dc77c19eb2df/payload-processing/0.log"
Sep 08 04:29:32.948330 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.948291    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"]
Sep 08 04:29:32.951647 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.951628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:32.954328 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.954303    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-520856\"/\"kube-root-ca.crt\""
Sep 08 04:29:32.955284 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.955262    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-520856\"/\"openshift-service-ca.crt\""
Sep 08 04:29:32.955418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.955398    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-520856\"/\"default-dockercfg-xhxs9\""
Sep 08 04:29:32.955494 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.955412    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-520856\"/\"ipp-route-520856-kserve-self-signed-certs\""
Sep 08 04:29:32.962563 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:32.962541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"]
Sep 08 04:29:33.102759 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.102716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-home\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.102878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.102784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-model-cache\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.102878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.102807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tls-certs\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.102878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.102831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-dshm\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.102878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.102854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmh5\" (UniqueName: \"kubernetes.io/projected/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-kube-api-access-5tmh5\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.103012 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.102905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tmp-dir\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.203931 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.203870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tmp-dir\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.203931 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.203925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-home\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.203960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-model-cache\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.203985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tls-certs\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.204024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-dshm\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.204047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmh5\" (UniqueName: \"kubernetes.io/projected/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-kube-api-access-5tmh5\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204315 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.204296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tmp-dir\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204374 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.204349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-home\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.204413 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.204391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-model-cache\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.206176 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.206148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-dshm\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.206576 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.206552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tls-certs\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.212938 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.212916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmh5\" (UniqueName: \"kubernetes.io/projected/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-kube-api-access-5tmh5\") pod \"ipp-route-520856-kserve-6c47dd756c-mkksd\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") " pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.261801 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.261780    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:33.535658 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.535554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"]
Sep 08 04:29:33.538499 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:29:33.538476    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc6b6733_7c3d_41f8_9a34_fcf6239ab5a4.slice/crio-192700fd67e6b8f13302480c7c0df7292987fb78fccc39273ab38f217ef0630e WatchSource:0}: Error finding container 192700fd67e6b8f13302480c7c0df7292987fb78fccc39273ab38f217ef0630e: Status 404 returned error can't find the container with id 192700fd67e6b8f13302480c7c0df7292987fb78fccc39273ab38f217ef0630e
Sep 08 04:29:33.630320 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.630296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd" event={"ID":"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4","Type":"ContainerStarted","Data":"fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d"}
Sep 08 04:29:33.630440 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.630337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd" event={"ID":"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4","Type":"ContainerStarted","Data":"192700fd67e6b8f13302480c7c0df7292987fb78fccc39273ab38f217ef0630e"}
Sep 08 04:29:33.650399 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:33.650349    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd" podStartSLOduration=1.65033502 podStartE2EDuration="1.65033502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:29:33.649283448 +0000 UTC m=+1754.007763009" watchObservedRunningTime="2026-09-08 04:29:33.65033502 +0000 UTC m=+1754.008814611"
Sep 08 04:29:38.448418 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:38.448372    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:29:43.262062 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:43.262028    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:43.262409 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:43.262075    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:43.273977 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:43.273951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:43.684631 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:43.684555    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:29:55.062098 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:29:55.062062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:30:00.150280 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.150251    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"]
Sep 08 04:30:00.156932 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.156905    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"
Sep 08 04:30:00.161144 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.161113    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\""
Sep 08 04:30:00.162670 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.162645    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"]
Sep 08 04:30:00.169234 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.169213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"
Sep 08 04:30:00.440368 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:30:00.440343    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b738ee_3690_4445_ad67_204ebb1ed349.slice/crio-bf8ff3d33ddb86d27d7331cefef57549605f253197cc1bf4249132c614127eac WatchSource:0}: Error finding container bf8ff3d33ddb86d27d7331cefef57549605f253197cc1bf4249132c614127eac: Status 404 returned error can't find the container with id bf8ff3d33ddb86d27d7331cefef57549605f253197cc1bf4249132c614127eac
Sep 08 04:30:00.440538 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.440516    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"]
Sep 08 04:30:00.749526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.749486    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q" event={"ID":"93b738ee-3690-4445-ad67-204ebb1ed349","Type":"ContainerStarted","Data":"343e5c684842c59becf6f323734eaeed7effbf4e7576458c75198eb442a370f2"}
Sep 08 04:30:00.749686 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.749544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q" event={"ID":"93b738ee-3690-4445-ad67-204ebb1ed349","Type":"ContainerStarted","Data":"bf8ff3d33ddb86d27d7331cefef57549605f253197cc1bf4249132c614127eac"}
Sep 08 04:30:00.755179 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.755160    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz_a5e595b1-dd15-4819-9294-dc77c19eb2df/payload-processing/0.log"
Sep 08 04:30:00.771548 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:00.771501    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q" podStartSLOduration=0.771486191 podStartE2EDuration="771.486191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:00.770044353 +0000 UTC m=+1781.128523912" watchObservedRunningTime="2026-09-08 04:30:00.771486191 +0000 UTC m=+1781.129965787"
Sep 08 04:30:01.756247 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:01.756209    2583 generic.go:356] "Generic (PLEG): container finished" podID="93b738ee-3690-4445-ad67-204ebb1ed349" containerID="343e5c684842c59becf6f323734eaeed7effbf4e7576458c75198eb442a370f2" exitCode=0
Sep 08 04:30:01.756634 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:01.756296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q" event={"ID":"93b738ee-3690-4445-ad67-204ebb1ed349","Type":"ContainerDied","Data":"343e5c684842c59becf6f323734eaeed7effbf4e7576458c75198eb442a370f2"}
Sep 08 04:30:02.875124 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:02.875100    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"
Sep 08 04:30:03.765703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:03.765670    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q" event={"ID":"93b738ee-3690-4445-ad67-204ebb1ed349","Type":"ContainerDied","Data":"bf8ff3d33ddb86d27d7331cefef57549605f253197cc1bf4249132c614127eac"}
Sep 08 04:30:03.765703 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:03.765682    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"
Sep 08 04:30:03.765955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:03.765717    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8ff3d33ddb86d27d7331cefef57549605f253197cc1bf4249132c614127eac"
Sep 08 04:30:14.943224 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:14.943145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:30:20.226031 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:20.226004    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:30:20.228336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:20.228312    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:30:42.556919 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:30:42.556877    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:31:03.406233 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.406197    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"]
Sep 08 04:31:03.406707 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.406407    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" podUID="e6374bc2-ef3a-45f2-8e53-9ddd146215d1" containerName="maas-api" containerID="cri-o://41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed" gracePeriod=30
Sep 08 04:31:03.445920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.445893    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"]
Sep 08 04:31:03.449368 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.449342    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-520856-29814030-wtj2q"]
Sep 08 04:31:03.479276 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.479252    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"]
Sep 08 04:31:03.479520 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.479491    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v" podUID="7673b720-b142-4b96-bc39-d4389f598dd7" containerName="payload-pre-processing" containerID="cri-o://d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410" gracePeriod=30
Sep 08 04:31:03.725483 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.725461    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:31:03.737588 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.737568    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:31:03.761209 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsb28\" (UniqueName: \"kubernetes.io/projected/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-kube-api-access-lsb28\") pod \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") "
Sep 08 04:31:03.761336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673b720-b142-4b96-bc39-d4389f598dd7-plugins-config-volume\") pod \"7673b720-b142-4b96-bc39-d4389f598dd7\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") "
Sep 08 04:31:03.761336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761232    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls\") pod \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") "
Sep 08 04:31:03.761336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761282    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-metrics-certs\") pod \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") "
Sep 08 04:31:03.761336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95pc9\" (UniqueName: \"kubernetes.io/projected/7673b720-b142-4b96-bc39-d4389f598dd7-kube-api-access-95pc9\") pod \"7673b720-b142-4b96-bc39-d4389f598dd7\" (UID: \"7673b720-b142-4b96-bc39-d4389f598dd7\") "
Sep 08 04:31:03.761336 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761321    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-openshift-service-ca\") pod \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\" (UID: \"e6374bc2-ef3a-45f2-8e53-9ddd146215d1\") "
Sep 08 04:31:03.761678 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7673b720-b142-4b96-bc39-d4389f598dd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7673b720-b142-4b96-bc39-d4389f598dd7" (UID: "7673b720-b142-4b96-bc39-d4389f598dd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:31:03.761972 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.761921    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6374bc2-ef3a-45f2-8e53-9ddd146215d1" (UID: "e6374bc2-ef3a-45f2-8e53-9ddd146215d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:31:03.763348 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.763321    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-kube-api-access-lsb28" (OuterVolumeSpecName: "kube-api-access-lsb28") pod "e6374bc2-ef3a-45f2-8e53-9ddd146215d1" (UID: "e6374bc2-ef3a-45f2-8e53-9ddd146215d1"). InnerVolumeSpecName "kube-api-access-lsb28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:31:03.763568 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.763546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6374bc2-ef3a-45f2-8e53-9ddd146215d1" (UID: "e6374bc2-ef3a-45f2-8e53-9ddd146215d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:31:03.763702 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.763683    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6374bc2-ef3a-45f2-8e53-9ddd146215d1" (UID: "e6374bc2-ef3a-45f2-8e53-9ddd146215d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:31:03.763784 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.763697    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7673b720-b142-4b96-bc39-d4389f598dd7-kube-api-access-95pc9" (OuterVolumeSpecName: "kube-api-access-95pc9") pod "7673b720-b142-4b96-bc39-d4389f598dd7" (UID: "7673b720-b142-4b96-bc39-d4389f598dd7"). InnerVolumeSpecName "kube-api-access-95pc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:31:03.862804 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.862778    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:03.862804 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.862802    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsb28\" (UniqueName: \"kubernetes.io/projected/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-kube-api-access-lsb28\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:03.862963 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.862812    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7673b720-b142-4b96-bc39-d4389f598dd7-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:03.862963 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.862821    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:03.862963 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.862828    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6374bc2-ef3a-45f2-8e53-9ddd146215d1-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:03.862963 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:03.862836    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95pc9\" (UniqueName: \"kubernetes.io/projected/7673b720-b142-4b96-bc39-d4389f598dd7-kube-api-access-95pc9\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:04.035662 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.035628    2583 generic.go:356] "Generic (PLEG): container finished" podID="e6374bc2-ef3a-45f2-8e53-9ddd146215d1" containerID="41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed" exitCode=0
Sep 08 04:31:04.035791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.035668    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"
Sep 08 04:31:04.035791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.035675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" event={"ID":"e6374bc2-ef3a-45f2-8e53-9ddd146215d1","Type":"ContainerDied","Data":"41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed"}
Sep 08 04:31:04.035791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.035728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj" event={"ID":"e6374bc2-ef3a-45f2-8e53-9ddd146215d1","Type":"ContainerDied","Data":"436c785869c9945641cd911a67756a478c4ec1b2e37e825e5bd3462a20277954"}
Sep 08 04:31:04.035791 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.035763    2583 scope.go:117] "RemoveContainer" containerID="41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed"
Sep 08 04:31:04.037311 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.037284    2583 generic.go:356] "Generic (PLEG): container finished" podID="7673b720-b142-4b96-bc39-d4389f598dd7" containerID="d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410" exitCode=0
Sep 08 04:31:04.037396 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.037329    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"
Sep 08 04:31:04.037445 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.037328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v" event={"ID":"7673b720-b142-4b96-bc39-d4389f598dd7","Type":"ContainerDied","Data":"d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410"}
Sep 08 04:31:04.037488 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.037462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v" event={"ID":"7673b720-b142-4b96-bc39-d4389f598dd7","Type":"ContainerDied","Data":"b49094173dc6a4a34009718a6fe23e226b830b27871c6abc9fe87c7c79b33253"}
Sep 08 04:31:04.046447 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.046433    2583 scope.go:117] "RemoveContainer" containerID="41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed"
Sep 08 04:31:04.046770 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:31:04.046724    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed\": container with ID starting with 41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed not found: ID does not exist" containerID="41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed"
Sep 08 04:31:04.046889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.046779    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed"} err="failed to get container status \"41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed\": rpc error: code = NotFound desc = could not find container \"41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed\": container with ID starting with 41468fccc364990e73195e066e4b1e0c6afa36776c3e8d37cadb24b6002a73ed not found: ID does not exist"
Sep 08 04:31:04.046889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.046802    2583 scope.go:117] "RemoveContainer" containerID="d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410"
Sep 08 04:31:04.054684 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.054669    2583 scope.go:117] "RemoveContainer" containerID="d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410"
Sep 08 04:31:04.054937 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:31:04.054919    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410\": container with ID starting with d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410 not found: ID does not exist" containerID="d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410"
Sep 08 04:31:04.054981 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.054943    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410"} err="failed to get container status \"d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410\": rpc error: code = NotFound desc = could not find container \"d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410\": container with ID starting with d8d1172121227c625b33f268428c8e59247512c2935f6f96091e0d5de22ee410 not found: ID does not exist"
Sep 08 04:31:04.070929 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.070907    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"]
Sep 08 04:31:04.073256 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.073235    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-520856-76dddcb89d-p56hj"]
Sep 08 04:31:04.100379 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.100360    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"]
Sep 08 04:31:04.110046 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.110022    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-520856-5cbf8fd7fd-7x77v"]
Sep 08 04:31:04.153423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.153404    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:31:04.158874 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.158853    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7673b720-b142-4b96-bc39-d4389f598dd7" path="/var/lib/kubelet/pods/7673b720-b142-4b96-bc39-d4389f598dd7/volumes"
Sep 08 04:31:04.159169 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.159155    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b738ee-3690-4445-ad67-204ebb1ed349" path="/var/lib/kubelet/pods/93b738ee-3690-4445-ad67-204ebb1ed349/volumes"
Sep 08 04:31:04.159513 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:04.159499    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6374bc2-ef3a-45f2-8e53-9ddd146215d1" path="/var/lib/kubelet/pods/e6374bc2-ef3a-45f2-8e53-9ddd146215d1/volumes"
Sep 08 04:31:09.038306 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:09.038268    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"]
Sep 08 04:31:14.050156 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:14.050121    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:31:20.440141 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:20.440109    2583 scope.go:117] "RemoveContainer" containerID="253ec82455519af66d848f98030177843e1f7fda71a1d393065205be1a956ffa"
Sep 08 04:31:24.057446 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.057405    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd" podUID="dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" containerName="main" containerID="cri-o://fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d" gracePeriod=45
Sep 08 04:31:24.293255 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.293228    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:31:24.425873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.425791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-model-cache\") pod \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") "
Sep 08 04:31:24.425873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.425846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-home\") pod \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") "
Sep 08 04:31:24.425873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.425871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmh5\" (UniqueName: \"kubernetes.io/projected/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-kube-api-access-5tmh5\") pod \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") "
Sep 08 04:31:24.426166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.425921    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tls-certs\") pod \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") "
Sep 08 04:31:24.426166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.425947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tmp-dir\") pod \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") "
Sep 08 04:31:24.426166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.425967    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-dshm\") pod \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\" (UID: \"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4\") "
Sep 08 04:31:24.426166 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.426076    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" (UID: "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:31:24.426362 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.426189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-home" (OuterVolumeSpecName: "home") pod "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" (UID: "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:31:24.426362 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.426201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" (UID: "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:31:24.426362 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.426234    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:24.428147 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.428117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" (UID: "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:31:24.428253 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.428192    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-kube-api-access-5tmh5" (OuterVolumeSpecName: "kube-api-access-5tmh5") pod "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" (UID: "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4"). InnerVolumeSpecName "kube-api-access-5tmh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:31:24.428253 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.428228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-dshm" (OuterVolumeSpecName: "dshm") pod "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" (UID: "dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:31:24.526728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.526692    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:24.526728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.526719    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:24.526728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.526728    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:24.526728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.526751    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:24.526994 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:24.526761    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmh5\" (UniqueName: \"kubernetes.io/projected/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4-kube-api-access-5tmh5\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:25.134194 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.134157    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" containerID="fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d" exitCode=0
Sep 08 04:31:25.134574 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.134213    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"
Sep 08 04:31:25.134574 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.134245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd" event={"ID":"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4","Type":"ContainerDied","Data":"fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d"}
Sep 08 04:31:25.134574 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.134293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd" event={"ID":"dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4","Type":"ContainerDied","Data":"192700fd67e6b8f13302480c7c0df7292987fb78fccc39273ab38f217ef0630e"}
Sep 08 04:31:25.134574 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.134312    2583 scope.go:117] "RemoveContainer" containerID="fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d"
Sep 08 04:31:25.143159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.143141    2583 scope.go:117] "RemoveContainer" containerID="fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d"
Sep 08 04:31:25.143462 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:31:25.143443    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d\": container with ID starting with fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d not found: ID does not exist" containerID="fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d"
Sep 08 04:31:25.143526 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.143470    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d"} err="failed to get container status \"fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d\": rpc error: code = NotFound desc = could not find container \"fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d\": container with ID starting with fb079dfa86cc29d8a68b420fe63d3d3d842fda7f693f220b40d9bc7aeddd6f8d not found: ID does not exist"
Sep 08 04:31:25.155471 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.155436    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"]
Sep 08 04:31:25.158854 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:25.158829    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-520856/ipp-route-520856-kserve-6c47dd756c-mkksd"]
Sep 08 04:31:26.160723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:26.160687    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" path="/var/lib/kubelet/pods/dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4/volumes"
Sep 08 04:31:59.185348 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.185267    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"]
Sep 08 04:31:59.185909 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.185481    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" podUID="bf90b524-ca36-4095-896b-181ed32293af" containerName="maas-api" containerID="cri-o://d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75" gracePeriod=30
Sep 08 04:31:59.262767 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.262727    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"]
Sep 08 04:31:59.262980 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.262941    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz" podUID="a5e595b1-dd15-4819-9294-dc77c19eb2df" containerName="payload-processing" containerID="cri-o://c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417" gracePeriod=30
Sep 08 04:31:59.285766 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.285727    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"]
Sep 08 04:31:59.285950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.285929    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv" podUID="eff90492-5078-4ad1-b931-3accc3ad2c15" containerName="payload-pre-processing" containerID="cri-o://bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c" gracePeriod=30
Sep 08 04:31:59.529631 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.529607    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:31:59.552192 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.552155    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:31:59.561370 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.561349    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:31:59.706908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.706837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e595b1-dd15-4819-9294-dc77c19eb2df-plugins-config-volume\") pod \"a5e595b1-dd15-4819-9294-dc77c19eb2df\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") "
Sep 08 04:31:59.706908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.706887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctldh\" (UniqueName: \"kubernetes.io/projected/a5e595b1-dd15-4819-9294-dc77c19eb2df-kube-api-access-ctldh\") pod \"a5e595b1-dd15-4819-9294-dc77c19eb2df\" (UID: \"a5e595b1-dd15-4819-9294-dc77c19eb2df\") "
Sep 08 04:31:59.707099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.706915    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b524-ca36-4095-896b-181ed32293af-openshift-service-ca\") pod \"bf90b524-ca36-4095-896b-181ed32293af\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") "
Sep 08 04:31:59.707099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.706941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mpsk\" (UniqueName: \"kubernetes.io/projected/eff90492-5078-4ad1-b931-3accc3ad2c15-kube-api-access-2mpsk\") pod \"eff90492-5078-4ad1-b931-3accc3ad2c15\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") "
Sep 08 04:31:59.707099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.706972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td9pw\" (UniqueName: \"kubernetes.io/projected/bf90b524-ca36-4095-896b-181ed32293af-kube-api-access-td9pw\") pod \"bf90b524-ca36-4095-896b-181ed32293af\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") "
Sep 08 04:31:59.707099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.706997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff90492-5078-4ad1-b931-3accc3ad2c15-plugins-config-volume\") pod \"eff90492-5078-4ad1-b931-3accc3ad2c15\" (UID: \"eff90492-5078-4ad1-b931-3accc3ad2c15\") "
Sep 08 04:31:59.707308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-metrics-certs\") pod \"bf90b524-ca36-4095-896b-181ed32293af\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") "
Sep 08 04:31:59.707308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls\") pod \"bf90b524-ca36-4095-896b-181ed32293af\" (UID: \"bf90b524-ca36-4095-896b-181ed32293af\") "
Sep 08 04:31:59.707308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e595b1-dd15-4819-9294-dc77c19eb2df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5e595b1-dd15-4819-9294-dc77c19eb2df" (UID: "a5e595b1-dd15-4819-9294-dc77c19eb2df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:31:59.707470 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707377    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf90b524-ca36-4095-896b-181ed32293af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf90b524-ca36-4095-896b-181ed32293af" (UID: "bf90b524-ca36-4095-896b-181ed32293af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:31:59.707470 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707393    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff90492-5078-4ad1-b931-3accc3ad2c15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eff90492-5078-4ad1-b931-3accc3ad2c15" (UID: "eff90492-5078-4ad1-b931-3accc3ad2c15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:31:59.707470 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707452    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff90492-5078-4ad1-b931-3accc3ad2c15-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.707615 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707469    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e595b1-dd15-4819-9294-dc77c19eb2df-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.707615 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.707487    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b524-ca36-4095-896b-181ed32293af-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.709239 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.709208    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf90b524-ca36-4095-896b-181ed32293af-kube-api-access-td9pw" (OuterVolumeSpecName: "kube-api-access-td9pw") pod "bf90b524-ca36-4095-896b-181ed32293af" (UID: "bf90b524-ca36-4095-896b-181ed32293af"). InnerVolumeSpecName "kube-api-access-td9pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:31:59.709371 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.709247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf90b524-ca36-4095-896b-181ed32293af" (UID: "bf90b524-ca36-4095-896b-181ed32293af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:31:59.709371 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.709307    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e595b1-dd15-4819-9294-dc77c19eb2df-kube-api-access-ctldh" (OuterVolumeSpecName: "kube-api-access-ctldh") pod "a5e595b1-dd15-4819-9294-dc77c19eb2df" (UID: "a5e595b1-dd15-4819-9294-dc77c19eb2df"). InnerVolumeSpecName "kube-api-access-ctldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:31:59.709547 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.709526    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff90492-5078-4ad1-b931-3accc3ad2c15-kube-api-access-2mpsk" (OuterVolumeSpecName: "kube-api-access-2mpsk") pod "eff90492-5078-4ad1-b931-3accc3ad2c15" (UID: "eff90492-5078-4ad1-b931-3accc3ad2c15"). InnerVolumeSpecName "kube-api-access-2mpsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:31:59.709590 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.709556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf90b524-ca36-4095-896b-181ed32293af" (UID: "bf90b524-ca36-4095-896b-181ed32293af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:31:59.808678 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.808657    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.808678 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.808677    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctldh\" (UniqueName: \"kubernetes.io/projected/a5e595b1-dd15-4819-9294-dc77c19eb2df-kube-api-access-ctldh\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.808813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.808687    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mpsk\" (UniqueName: \"kubernetes.io/projected/eff90492-5078-4ad1-b931-3accc3ad2c15-kube-api-access-2mpsk\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.808813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.808697    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9pw\" (UniqueName: \"kubernetes.io/projected/bf90b524-ca36-4095-896b-181ed32293af-kube-api-access-td9pw\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:31:59.808813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:31:59.808705    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf90b524-ca36-4095-896b-181ed32293af-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:32:00.289620 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.289584    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5e595b1-dd15-4819-9294-dc77c19eb2df" containerID="c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417" exitCode=0
Sep 08 04:32:00.289973 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.289628    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"
Sep 08 04:32:00.289973 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.289665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz" event={"ID":"a5e595b1-dd15-4819-9294-dc77c19eb2df","Type":"ContainerDied","Data":"c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417"}
Sep 08 04:32:00.289973 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.289712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz" event={"ID":"a5e595b1-dd15-4819-9294-dc77c19eb2df","Type":"ContainerDied","Data":"e10e420223de5a4820933a241747a36e649a7abb57cd5ea8cfa8478d29da22e6"}
Sep 08 04:32:00.289973 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.289758    2583 scope.go:117] "RemoveContainer" containerID="c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417"
Sep 08 04:32:00.291208 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.291183    2583 generic.go:356] "Generic (PLEG): container finished" podID="eff90492-5078-4ad1-b931-3accc3ad2c15" containerID="bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c" exitCode=0
Sep 08 04:32:00.291345 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.291226    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"
Sep 08 04:32:00.291345 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.291283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv" event={"ID":"eff90492-5078-4ad1-b931-3accc3ad2c15","Type":"ContainerDied","Data":"bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c"}
Sep 08 04:32:00.291345 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.291331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv" event={"ID":"eff90492-5078-4ad1-b931-3accc3ad2c15","Type":"ContainerDied","Data":"c80b16bef6930ae3b619f782edec058af096ef83ee4222690f672ec2ac195656"}
Sep 08 04:32:00.292902 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.292866    2583 generic.go:356] "Generic (PLEG): container finished" podID="bf90b524-ca36-4095-896b-181ed32293af" containerID="d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75" exitCode=0
Sep 08 04:32:00.293208 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.292918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" event={"ID":"bf90b524-ca36-4095-896b-181ed32293af","Type":"ContainerDied","Data":"d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75"}
Sep 08 04:32:00.293208 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.292958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv" event={"ID":"bf90b524-ca36-4095-896b-181ed32293af","Type":"ContainerDied","Data":"938846d6e74b71807215813a8405340b2fbeee3b17238c664faa4b60523f28c0"}
Sep 08 04:32:00.293208 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.292998    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"
Sep 08 04:32:00.298853 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.298837    2583 scope.go:117] "RemoveContainer" containerID="c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417"
Sep 08 04:32:00.299136 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:32:00.299115    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417\": container with ID starting with c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417 not found: ID does not exist" containerID="c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417"
Sep 08 04:32:00.299212 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.299148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417"} err="failed to get container status \"c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417\": rpc error: code = NotFound desc = could not find container \"c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417\": container with ID starting with c2ffe7e6537543bc11e97b3b414f8ebb7a2824af509d3b2f7896159f792a3417 not found: ID does not exist"
Sep 08 04:32:00.299212 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.299171    2583 scope.go:117] "RemoveContainer" containerID="bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c"
Sep 08 04:32:00.306326 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.306309    2583 scope.go:117] "RemoveContainer" containerID="bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c"
Sep 08 04:32:00.306553 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:32:00.306535    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c\": container with ID starting with bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c not found: ID does not exist" containerID="bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c"
Sep 08 04:32:00.306595 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.306559    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c"} err="failed to get container status \"bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c\": rpc error: code = NotFound desc = could not find container \"bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c\": container with ID starting with bd345a6dad86bd6e4f46a8d89f730d91fa580cf8afc5b11b64d96659fb105a2c not found: ID does not exist"
Sep 08 04:32:00.306595 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.306574    2583 scope.go:117] "RemoveContainer" containerID="d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75"
Sep 08 04:32:00.311159 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.311134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"]
Sep 08 04:32:00.313950 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.313935    2583 scope.go:117] "RemoveContainer" containerID="d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75"
Sep 08 04:32:00.314239 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:32:00.314220    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75\": container with ID starting with d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75 not found: ID does not exist" containerID="d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75"
Sep 08 04:32:00.314282 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.314246    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75"} err="failed to get container status \"d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75\": rpc error: code = NotFound desc = could not find container \"d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75\": container with ID starting with d644bea7afe8c128224a086454030a374534e5f83f96c2c86f5e1719efddbd75 not found: ID does not exist"
Sep 08 04:32:00.317061 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.317040    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6c84ae-6b9bffbfc4-zmcpv"]
Sep 08 04:32:00.327677 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.327656    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"]
Sep 08 04:32:00.331359 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.331338    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6c84ae-cd44c5dc7-srkhz"]
Sep 08 04:32:00.341841 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.341821    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"]
Sep 08 04:32:00.345472 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:00.345451    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6c84ae-54d4df466d-l59nv"]
Sep 08 04:32:02.158646 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:02.158613    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e595b1-dd15-4819-9294-dc77c19eb2df" path="/var/lib/kubelet/pods/a5e595b1-dd15-4819-9294-dc77c19eb2df/volumes"
Sep 08 04:32:02.159022 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:02.158982    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf90b524-ca36-4095-896b-181ed32293af" path="/var/lib/kubelet/pods/bf90b524-ca36-4095-896b-181ed32293af/volumes"
Sep 08 04:32:02.159283 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:02.159269    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff90492-5078-4ad1-b931-3accc3ad2c15" path="/var/lib/kubelet/pods/eff90492-5078-4ad1-b931-3accc3ad2c15/volumes"
Sep 08 04:32:02.555856 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:02.555831    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:32:06.369996 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.369962    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"]
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" containerName="main"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370278    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" containerName="main"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370289    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b738ee-3690-4445-ad67-204ebb1ed349" containerName="cleanup"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370294    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b738ee-3690-4445-ad67-204ebb1ed349" containerName="cleanup"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370311    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf90b524-ca36-4095-896b-181ed32293af" containerName="maas-api"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370316    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf90b524-ca36-4095-896b-181ed32293af" containerName="maas-api"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370325    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6374bc2-ef3a-45f2-8e53-9ddd146215d1" containerName="maas-api"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370330    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6374bc2-ef3a-45f2-8e53-9ddd146215d1" containerName="maas-api"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370338    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7673b720-b142-4b96-bc39-d4389f598dd7" containerName="payload-pre-processing"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7673b720-b142-4b96-bc39-d4389f598dd7" containerName="payload-pre-processing"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370348    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff90492-5078-4ad1-b931-3accc3ad2c15" containerName="payload-pre-processing"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370353    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff90492-5078-4ad1-b931-3accc3ad2c15" containerName="payload-pre-processing"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370359    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e595b1-dd15-4819-9294-dc77c19eb2df" containerName="payload-processing"
Sep 08 04:32:06.370380 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370364    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e595b1-dd15-4819-9294-dc77c19eb2df" containerName="payload-processing"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370408    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff90492-5078-4ad1-b931-3accc3ad2c15" containerName="payload-pre-processing"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370416    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf90b524-ca36-4095-896b-181ed32293af" containerName="maas-api"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370423    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7673b720-b142-4b96-bc39-d4389f598dd7" containerName="payload-pre-processing"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370430    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6374bc2-ef3a-45f2-8e53-9ddd146215d1" containerName="maas-api"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370436    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6b6733-7c3d-41f8-9a34-fcf6239ab5a4" containerName="main"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370442    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e595b1-dd15-4819-9294-dc77c19eb2df" containerName="payload-processing"
Sep 08 04:32:06.370880 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.370446    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b738ee-3690-4445-ad67-204ebb1ed349" containerName="cleanup"
Sep 08 04:32:06.374951 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.374933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:06.377230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.377207    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:32:06.377361 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.377248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\""
Sep 08 04:32:06.381087 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.381064    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"]
Sep 08 04:32:06.454034 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.454006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06460dc5-5c40-40fb-bbd5-85550374be68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78\" (UID: \"06460dc5-5c40-40fb-bbd5-85550374be68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:06.555061 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.555026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06460dc5-5c40-40fb-bbd5-85550374be68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78\" (UID: \"06460dc5-5c40-40fb-bbd5-85550374be68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:06.555569 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.555551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06460dc5-5c40-40fb-bbd5-85550374be68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78\" (UID: \"06460dc5-5c40-40fb-bbd5-85550374be68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:06.686403 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.686350    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:06.953598 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:06.953574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"]
Sep 08 04:32:06.956077 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:32:06.956052    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06460dc5_5c40_40fb_bbd5_85550374be68.slice/crio-bf52f13c97e5d80dc59290ff0efc3da495e3944c7f914d4c1e0b30b65bd96eaf WatchSource:0}: Error finding container bf52f13c97e5d80dc59290ff0efc3da495e3944c7f914d4c1e0b30b65bd96eaf: Status 404 returned error can't find the container with id bf52f13c97e5d80dc59290ff0efc3da495e3944c7f914d4c1e0b30b65bd96eaf
Sep 08 04:32:07.324556 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:07.324533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78" event={"ID":"06460dc5-5c40-40fb-bbd5-85550374be68","Type":"ContainerStarted","Data":"09c7e38ee304a54a6835dc74c00208592bba4cf01a8a4d41270405068ae32f25"}
Sep 08 04:32:07.324665 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:07.324576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78" event={"ID":"06460dc5-5c40-40fb-bbd5-85550374be68","Type":"ContainerStarted","Data":"bf52f13c97e5d80dc59290ff0efc3da495e3944c7f914d4c1e0b30b65bd96eaf"}
Sep 08 04:32:07.339627 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:07.339584    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78" podStartSLOduration=1.339570435 podStartE2EDuration="1.339570435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:32:07.337633171 +0000 UTC m=+1907.696112730" watchObservedRunningTime="2026-09-08 04:32:07.339570435 +0000 UTC m=+1907.698049994"
Sep 08 04:32:08.329755 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:08.329710    2583 generic.go:356] "Generic (PLEG): container finished" podID="06460dc5-5c40-40fb-bbd5-85550374be68" containerID="09c7e38ee304a54a6835dc74c00208592bba4cf01a8a4d41270405068ae32f25" exitCode=0
Sep 08 04:32:08.330071 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:08.329782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78" event={"ID":"06460dc5-5c40-40fb-bbd5-85550374be68","Type":"ContainerDied","Data":"09c7e38ee304a54a6835dc74c00208592bba4cf01a8a4d41270405068ae32f25"}
Sep 08 04:32:09.456623 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:09.456599    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:09.475307 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:09.475286    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06460dc5-5c40-40fb-bbd5-85550374be68-maas-api-service-ca\") pod \"06460dc5-5c40-40fb-bbd5-85550374be68\" (UID: \"06460dc5-5c40-40fb-bbd5-85550374be68\") "
Sep 08 04:32:09.475610 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:09.475589    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06460dc5-5c40-40fb-bbd5-85550374be68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06460dc5-5c40-40fb-bbd5-85550374be68" (UID: "06460dc5-5c40-40fb-bbd5-85550374be68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:32:09.575712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:09.575694    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06460dc5-5c40-40fb-bbd5-85550374be68-maas-api-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:32:10.339626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:10.339593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78" event={"ID":"06460dc5-5c40-40fb-bbd5-85550374be68","Type":"ContainerDied","Data":"bf52f13c97e5d80dc59290ff0efc3da495e3944c7f914d4c1e0b30b65bd96eaf"}
Sep 08 04:32:10.339773 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:10.339640    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf52f13c97e5d80dc59290ff0efc3da495e3944c7f914d4c1e0b30b65bd96eaf"
Sep 08 04:32:10.339773 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:10.339603    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"
Sep 08 04:32:16.952990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:16.952956    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"]
Sep 08 04:32:16.953479 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:16.953240    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87" podUID="6ede1a61-2715-4569-9193-ef8b674dd085" containerName="payload-pre-processing" containerID="cri-o://a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3" gracePeriod=30
Sep 08 04:32:17.152903 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.152869    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"]
Sep 08 04:32:17.155578 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.155551    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2a8bb-2rm78"]
Sep 08 04:32:17.196004 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.195985    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:32:17.229050 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.228991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwg22\" (UniqueName: \"kubernetes.io/projected/6ede1a61-2715-4569-9193-ef8b674dd085-kube-api-access-rwg22\") pod \"6ede1a61-2715-4569-9193-ef8b674dd085\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") "
Sep 08 04:32:17.229050 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.229041    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ede1a61-2715-4569-9193-ef8b674dd085-plugins-config-volume\") pod \"6ede1a61-2715-4569-9193-ef8b674dd085\" (UID: \"6ede1a61-2715-4569-9193-ef8b674dd085\") "
Sep 08 04:32:17.229404 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.229383    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ede1a61-2715-4569-9193-ef8b674dd085-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ede1a61-2715-4569-9193-ef8b674dd085" (UID: "6ede1a61-2715-4569-9193-ef8b674dd085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:32:17.230990 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.230965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ede1a61-2715-4569-9193-ef8b674dd085-kube-api-access-rwg22" (OuterVolumeSpecName: "kube-api-access-rwg22") pod "6ede1a61-2715-4569-9193-ef8b674dd085" (UID: "6ede1a61-2715-4569-9193-ef8b674dd085"). InnerVolumeSpecName "kube-api-access-rwg22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:32:17.330128 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.330102    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwg22\" (UniqueName: \"kubernetes.io/projected/6ede1a61-2715-4569-9193-ef8b674dd085-kube-api-access-rwg22\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:32:17.330128 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.330126    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ede1a61-2715-4569-9193-ef8b674dd085-plugins-config-volume\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:32:17.369341 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.369316    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ede1a61-2715-4569-9193-ef8b674dd085" containerID="a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3" exitCode=0
Sep 08 04:32:17.369435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.369360    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"
Sep 08 04:32:17.369435 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.369399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87" event={"ID":"6ede1a61-2715-4569-9193-ef8b674dd085","Type":"ContainerDied","Data":"a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3"}
Sep 08 04:32:17.369522 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.369466    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87" event={"ID":"6ede1a61-2715-4569-9193-ef8b674dd085","Type":"ContainerDied","Data":"979db62c54f0485b7490293415ede579342fdd014807d60367e741c38db35685"}
Sep 08 04:32:17.369522 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.369488    2583 scope.go:117] "RemoveContainer" containerID="a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3"
Sep 08 04:32:17.378185 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.378169    2583 scope.go:117] "RemoveContainer" containerID="a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3"
Sep 08 04:32:17.378472 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:32:17.378452    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3\": container with ID starting with a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3 not found: ID does not exist" containerID="a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3"
Sep 08 04:32:17.378536 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.378479    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3"} err="failed to get container status \"a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3\": rpc error: code = NotFound desc = could not find container \"a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3\": container with ID starting with a610d376d7910ded686b85cff85a146f8a7eac588bd2f58a5471acaa0ce107c3 not found: ID does not exist"
Sep 08 04:32:17.402562 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.402542    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"]
Sep 08 04:32:17.431206 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:17.431185    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c2a8bb-f54b96578-mlv87"]
Sep 08 04:32:18.159444 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:18.159416    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06460dc5-5c40-40fb-bbd5-85550374be68" path="/var/lib/kubelet/pods/06460dc5-5c40-40fb-bbd5-85550374be68/volumes"
Sep 08 04:32:18.159780 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:18.159732    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ede1a61-2715-4569-9193-ef8b674dd085" path="/var/lib/kubelet/pods/6ede1a61-2715-4569-9193-ef8b674dd085/volumes"
Sep 08 04:32:20.464948 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:20.464918    2583 scope.go:117] "RemoveContainer" containerID="b5275862840197ed0efb0bf570a0e148bcdf5879714aab19cfcd020d8e16ff6b"
Sep 08 04:32:20.659774 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:20.659750    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:32:24.235149 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235117    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"]
Sep 08 04:32:24.235487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235386    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ede1a61-2715-4569-9193-ef8b674dd085" containerName="payload-pre-processing"
Sep 08 04:32:24.235487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235396    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ede1a61-2715-4569-9193-ef8b674dd085" containerName="payload-pre-processing"
Sep 08 04:32:24.235487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235413    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06460dc5-5c40-40fb-bbd5-85550374be68" containerName="revoke-keys"
Sep 08 04:32:24.235487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235419    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="06460dc5-5c40-40fb-bbd5-85550374be68" containerName="revoke-keys"
Sep 08 04:32:24.235487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235466    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ede1a61-2715-4569-9193-ef8b674dd085" containerName="payload-pre-processing"
Sep 08 04:32:24.235487 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.235475    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="06460dc5-5c40-40fb-bbd5-85550374be68" containerName="revoke-keys"
Sep 08 04:32:24.238194 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.238177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:24.240659 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.240639    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\""
Sep 08 04:32:24.240813 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.240700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:32:24.245267 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.245243    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"]
Sep 08 04:32:24.279733 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.279708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ac07-b279-486e-9937-0edd4adc746a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj\" (UID: \"69f7ac07-b279-486e-9937-0edd4adc746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:24.380576 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.380554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ac07-b279-486e-9937-0edd4adc746a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj\" (UID: \"69f7ac07-b279-486e-9937-0edd4adc746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:24.381128 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.381109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ac07-b279-486e-9937-0edd4adc746a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj\" (UID: \"69f7ac07-b279-486e-9937-0edd4adc746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:24.548905 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.548875    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:24.820878 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:24.820853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"]
Sep 08 04:32:24.823273 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:32:24.823242    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f7ac07_b279_486e_9937_0edd4adc746a.slice/crio-941782be34fb0bcaec80de5bad7d93bf8e7f4cd29123aadb28082626631c1166 WatchSource:0}: Error finding container 941782be34fb0bcaec80de5bad7d93bf8e7f4cd29123aadb28082626631c1166: Status 404 returned error can't find the container with id 941782be34fb0bcaec80de5bad7d93bf8e7f4cd29123aadb28082626631c1166
Sep 08 04:32:25.406339 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:25.406306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj" event={"ID":"69f7ac07-b279-486e-9937-0edd4adc746a","Type":"ContainerStarted","Data":"fff8d1b17536483084c6b2dcfe0946b8f646c9a3bb7330841ac0c0bc2f9015d7"}
Sep 08 04:32:25.406688 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:25.406355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj" event={"ID":"69f7ac07-b279-486e-9937-0edd4adc746a","Type":"ContainerStarted","Data":"941782be34fb0bcaec80de5bad7d93bf8e7f4cd29123aadb28082626631c1166"}
Sep 08 04:32:25.420895 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:25.420848    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj" podStartSLOduration=1.420812696 podStartE2EDuration="1.420812696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:32:25.419993761 +0000 UTC m=+1925.778473330" watchObservedRunningTime="2026-09-08 04:32:25.420812696 +0000 UTC m=+1925.779292278"
Sep 08 04:32:26.412125 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:26.412093    2583 generic.go:356] "Generic (PLEG): container finished" podID="69f7ac07-b279-486e-9937-0edd4adc746a" containerID="fff8d1b17536483084c6b2dcfe0946b8f646c9a3bb7330841ac0c0bc2f9015d7" exitCode=0
Sep 08 04:32:26.412125 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:26.412129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj" event={"ID":"69f7ac07-b279-486e-9937-0edd4adc746a","Type":"ContainerDied","Data":"fff8d1b17536483084c6b2dcfe0946b8f646c9a3bb7330841ac0c0bc2f9015d7"}
Sep 08 04:32:27.535842 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:27.535822    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:27.605016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:27.604991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ac07-b279-486e-9937-0edd4adc746a-maas-api-service-ca\") pod \"69f7ac07-b279-486e-9937-0edd4adc746a\" (UID: \"69f7ac07-b279-486e-9937-0edd4adc746a\") "
Sep 08 04:32:27.605288 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:27.605267    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f7ac07-b279-486e-9937-0edd4adc746a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69f7ac07-b279-486e-9937-0edd4adc746a" (UID: "69f7ac07-b279-486e-9937-0edd4adc746a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:32:27.706016 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:27.705967    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ac07-b279-486e-9937-0edd4adc746a-maas-api-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:32:28.420875 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:28.420809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj" event={"ID":"69f7ac07-b279-486e-9937-0edd4adc746a","Type":"ContainerDied","Data":"941782be34fb0bcaec80de5bad7d93bf8e7f4cd29123aadb28082626631c1166"}
Sep 08 04:32:28.420875 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:28.420845    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"
Sep 08 04:32:28.420875 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:28.420862    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941782be34fb0bcaec80de5bad7d93bf8e7f4cd29123aadb28082626631c1166"
Sep 08 04:32:35.035006 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:35.034975    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"]
Sep 08 04:32:35.038255 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:35.038230    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-720789-fkqgj"]
Sep 08 04:32:36.159645 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:36.159611    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f7ac07-b279-486e-9937-0edd4adc746a" path="/var/lib/kubelet/pods/69f7ac07-b279-486e-9937-0edd4adc746a/volumes"
Sep 08 04:32:38.058085 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:38.058052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:32:46.545902 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:32:46.545864    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:34:07.542829 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:34:07.542796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:34:34.250927 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:34:34.250890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:34:41.255334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:34:41.255296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:34:48.728664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:34:48.728583    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:34:56.251606 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:34:56.251572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:35:10.943734 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:35:10.943704    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:35:19.145476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:35:19.145439    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:35:20.249783 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:35:20.249752    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:35:20.253126 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:35:20.253104    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:36:20.514282 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:36:20.514255    2583 scope.go:117] "RemoveContainer" containerID="343e5c684842c59becf6f323734eaeed7effbf4e7576458c75198eb442a370f2"
Sep 08 04:36:48.852446 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:36:48.852411    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:36:57.154514 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:36:57.154469    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:37:05.547103 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:05.547069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:37:20.340980 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.340949    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"]
Sep 08 04:37:20.341400 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.341198    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f7ac07-b279-486e-9937-0edd4adc746a" containerName="revoke-keys"
Sep 08 04:37:20.341400 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.341209    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f7ac07-b279-486e-9937-0edd4adc746a" containerName="revoke-keys"
Sep 08 04:37:20.341400 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.341270    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f7ac07-b279-486e-9937-0edd4adc746a" containerName="revoke-keys"
Sep 08 04:37:20.344055 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.344038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.346328 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.346301    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e4456688-kserve-self-signed-certs\""
Sep 08 04:37:20.354056 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.354028    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"]
Sep 08 04:37:20.461324 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.461301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tmp-dir\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.461450 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.461340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-model-cache\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.461450 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.461358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52v5\" (UniqueName: \"kubernetes.io/projected/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-kube-api-access-d52v5\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.461450 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.461428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tls-certs\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.461557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.461460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-home\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.461557 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.461485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-dshm\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562461 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-model-cache\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52v5\" (UniqueName: \"kubernetes.io/projected/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-kube-api-access-d52v5\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tls-certs\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-home\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-dshm\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562809 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tmp-dir\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562907 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-model-cache\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.562975 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-home\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.563017 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.562976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tmp-dir\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.564601 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.564584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-dshm\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.565008 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.564987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tls-certs\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.571014 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.570993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52v5\" (UniqueName: \"kubernetes.io/projected/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-kube-api-access-d52v5\") pod \"e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") " pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.652621 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.652567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:20.921231 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:20.921208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"]
Sep 08 04:37:20.923769 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:37:20.923732    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36ede9e8_f33a_4f06_8b27_5f5f49f7c613.slice/crio-8fb94cbaf8cf38806baf16164cd918ddacd12d1450097d3d008f1033be1fb60c WatchSource:0}: Error finding container 8fb94cbaf8cf38806baf16164cd918ddacd12d1450097d3d008f1033be1fb60c: Status 404 returned error can't find the container with id 8fb94cbaf8cf38806baf16164cd918ddacd12d1450097d3d008f1033be1fb60c
Sep 08 04:37:21.612471 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:21.612437    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h" event={"ID":"36ede9e8-f33a-4f06-8b27-5f5f49f7c613","Type":"ContainerStarted","Data":"c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3"}
Sep 08 04:37:21.612843 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:21.612481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h" event={"ID":"36ede9e8-f33a-4f06-8b27-5f5f49f7c613","Type":"ContainerStarted","Data":"8fb94cbaf8cf38806baf16164cd918ddacd12d1450097d3d008f1033be1fb60c"}
Sep 08 04:37:21.630057 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:21.629996    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h" podStartSLOduration=1.629975236 podStartE2EDuration="1.629975236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:37:21.629643834 +0000 UTC m=+2221.988123394" watchObservedRunningTime="2026-09-08 04:37:21.629975236 +0000 UTC m=+2221.988454801"
Sep 08 04:37:23.156504 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:23.156458    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:37:30.652799 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:30.652761    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:30.653230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:30.652809    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:30.664818 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:30.664794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:31.669474 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:31.669444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:37:38.825693 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.825663    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"]
Sep 08 04:37:38.828682 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.828663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.830873 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.830852    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e4456688-kserve-self-signed-certs\""
Sep 08 04:37:38.837632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.837607    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"]
Sep 08 04:37:38.887373 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.887347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-dshm\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.887468 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.887390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-home\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.887468 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.887412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tmp-dir\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.887468 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.887429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-model-cache\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.887586 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.887479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tls-certs\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.887586 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.887522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6flb\" (UniqueName: \"kubernetes.io/projected/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-kube-api-access-m6flb\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.988724 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.988698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-dshm\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.988826 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.988758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-home\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.988826 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.988783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tmp-dir\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.988826 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.988808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-model-cache\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.988955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.988835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tls-certs\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.988955 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.988867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6flb\" (UniqueName: \"kubernetes.io/projected/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-kube-api-access-m6flb\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.989172 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.989149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tmp-dir\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.989244 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.989178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-home\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.989244 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.989228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-model-cache\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.990693 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.990674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-dshm\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.991137 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.991121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tls-certs\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:38.996276 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:38.996259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6flb\" (UniqueName: \"kubernetes.io/projected/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-kube-api-access-m6flb\") pod \"e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") " pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:39.138099 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:39.138050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:39.406382 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:39.406302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"]
Sep 08 04:37:39.409015 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:37:39.408985    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a41ec6f_c903_4e2b_a7bd_649838af3c4f.slice/crio-933707f05952741cf5cf4d1c9dc077a85d9dd57709f64e9dd1ef9f67bdcf704c WatchSource:0}: Error finding container 933707f05952741cf5cf4d1c9dc077a85d9dd57709f64e9dd1ef9f67bdcf704c: Status 404 returned error can't find the container with id 933707f05952741cf5cf4d1c9dc077a85d9dd57709f64e9dd1ef9f67bdcf704c
Sep 08 04:37:39.690260 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:39.690187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z" event={"ID":"5a41ec6f-c903-4e2b-a7bd-649838af3c4f","Type":"ContainerStarted","Data":"944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5"}
Sep 08 04:37:39.690260 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:39.690229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z" event={"ID":"5a41ec6f-c903-4e2b-a7bd-649838af3c4f","Type":"ContainerStarted","Data":"933707f05952741cf5cf4d1c9dc077a85d9dd57709f64e9dd1ef9f67bdcf704c"}
Sep 08 04:37:39.710037 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:39.709997    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z" podStartSLOduration=1.70998034 podStartE2EDuration="1.70998034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:37:39.709122463 +0000 UTC m=+2240.067602021" watchObservedRunningTime="2026-09-08 04:37:39.70998034 +0000 UTC m=+2240.068459900"
Sep 08 04:37:42.445040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:42.445005    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:37:49.138976 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:49.138938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:49.139427 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:49.139012    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:49.150921 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:49.150897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:37:49.741715 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:37:49.741684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:38:09.288712 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:09.288677    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"]
Sep 08 04:38:10.548399 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:10.548364    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"]
Sep 08 04:38:12.552020 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:12.551986    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:38:20.536701 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:20.536671    2583 scope.go:117] "RemoveContainer" containerID="09c7e38ee304a54a6835dc74c00208592bba4cf01a8a4d41270405068ae32f25"
Sep 08 04:38:20.648066 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:20.648028    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:38:24.306750 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.306703    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h" podUID="36ede9e8-f33a-4f06-8b27-5f5f49f7c613" containerName="main" containerID="cri-o://c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3" gracePeriod=45
Sep 08 04:38:24.534664 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.534642    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:38:24.697428 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697362    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tmp-dir\") pod \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") "
Sep 08 04:38:24.697428 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697406    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d52v5\" (UniqueName: \"kubernetes.io/projected/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-kube-api-access-d52v5\") pod \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") "
Sep 08 04:38:24.697428 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697434    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-model-cache\") pod \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") "
Sep 08 04:38:24.697633 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697450    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-dshm\") pod \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") "
Sep 08 04:38:24.697633 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697531    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tls-certs\") pod \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") "
Sep 08 04:38:24.697633 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697587    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-home\") pod \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\" (UID: \"36ede9e8-f33a-4f06-8b27-5f5f49f7c613\") "
Sep 08 04:38:24.697633 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-model-cache" (OuterVolumeSpecName: "model-cache") pod "36ede9e8-f33a-4f06-8b27-5f5f49f7c613" (UID: "36ede9e8-f33a-4f06-8b27-5f5f49f7c613"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:24.697959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697635    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36ede9e8-f33a-4f06-8b27-5f5f49f7c613" (UID: "36ede9e8-f33a-4f06-8b27-5f5f49f7c613"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:24.697959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697854    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:24.697959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697869    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:24.697959 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.697944    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-home" (OuterVolumeSpecName: "home") pod "36ede9e8-f33a-4f06-8b27-5f5f49f7c613" (UID: "36ede9e8-f33a-4f06-8b27-5f5f49f7c613"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:24.699494 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.699474    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-kube-api-access-d52v5" (OuterVolumeSpecName: "kube-api-access-d52v5") pod "36ede9e8-f33a-4f06-8b27-5f5f49f7c613" (UID: "36ede9e8-f33a-4f06-8b27-5f5f49f7c613"). InnerVolumeSpecName "kube-api-access-d52v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:38:24.699579 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.699473    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-dshm" (OuterVolumeSpecName: "dshm") pod "36ede9e8-f33a-4f06-8b27-5f5f49f7c613" (UID: "36ede9e8-f33a-4f06-8b27-5f5f49f7c613"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:24.699579 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.699473    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36ede9e8-f33a-4f06-8b27-5f5f49f7c613" (UID: "36ede9e8-f33a-4f06-8b27-5f5f49f7c613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:38:24.798860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.798837    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:24.798860 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.798856    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52v5\" (UniqueName: \"kubernetes.io/projected/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-kube-api-access-d52v5\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:24.798994 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.798866    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:24.798994 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.798874    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ede9e8-f33a-4f06-8b27-5f5f49f7c613-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:24.875844 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.875817    2583 generic.go:356] "Generic (PLEG): container finished" podID="36ede9e8-f33a-4f06-8b27-5f5f49f7c613" containerID="c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3" exitCode=0
Sep 08 04:38:24.875936 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.875855    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"
Sep 08 04:38:24.875936 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.875902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h" event={"ID":"36ede9e8-f33a-4f06-8b27-5f5f49f7c613","Type":"ContainerDied","Data":"c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3"}
Sep 08 04:38:24.876021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.875939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h" event={"ID":"36ede9e8-f33a-4f06-8b27-5f5f49f7c613","Type":"ContainerDied","Data":"8fb94cbaf8cf38806baf16164cd918ddacd12d1450097d3d008f1033be1fb60c"}
Sep 08 04:38:24.876021 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.875956    2583 scope.go:117] "RemoveContainer" containerID="c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3"
Sep 08 04:38:24.884323 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.884311    2583 scope.go:117] "RemoveContainer" containerID="c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3"
Sep 08 04:38:24.884585 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:38:24.884568    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3\": container with ID starting with c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3 not found: ID does not exist" containerID="c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3"
Sep 08 04:38:24.884628 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.884592    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3"} err="failed to get container status \"c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3\": rpc error: code = NotFound desc = could not find container \"c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3\": container with ID starting with c439bc94c645105067e3d7db6c1fc0b64c20c63feeec95b6aba5a58be6caead3 not found: ID does not exist"
Sep 08 04:38:24.895193 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.895174    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"]
Sep 08 04:38:24.897271 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:24.897254    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e4456688-kserve-85b89c6bf5-mzp2h"]
Sep 08 04:38:25.566855 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.566817    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z" podUID="5a41ec6f-c903-4e2b-a7bd-649838af3c4f" containerName="main" containerID="cri-o://944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5" gracePeriod=45
Sep 08 04:38:25.808495 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.808472    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:38:25.882230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.882166    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a41ec6f-c903-4e2b-a7bd-649838af3c4f" containerID="944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5" exitCode=0
Sep 08 04:38:25.882230 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.882205    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"
Sep 08 04:38:25.882385 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.882240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z" event={"ID":"5a41ec6f-c903-4e2b-a7bd-649838af3c4f","Type":"ContainerDied","Data":"944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5"}
Sep 08 04:38:25.882385 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.882280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z" event={"ID":"5a41ec6f-c903-4e2b-a7bd-649838af3c4f","Type":"ContainerDied","Data":"933707f05952741cf5cf4d1c9dc077a85d9dd57709f64e9dd1ef9f67bdcf704c"}
Sep 08 04:38:25.882385 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.882297    2583 scope.go:117] "RemoveContainer" containerID="944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5"
Sep 08 04:38:25.889629 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.889603    2583 scope.go:117] "RemoveContainer" containerID="944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5"
Sep 08 04:38:25.889897 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:38:25.889879    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5\": container with ID starting with 944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5 not found: ID does not exist" containerID="944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5"
Sep 08 04:38:25.889949 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.889907    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5"} err="failed to get container status \"944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5\": rpc error: code = NotFound desc = could not find container \"944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5\": container with ID starting with 944f8360d2daf3b27b58aa02dcab6ba86a6556e2721c2d91ce7b9d94cd8ebab5 not found: ID does not exist"
Sep 08 04:38:25.906163 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tmp-dir\") pod \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") "
Sep 08 04:38:25.906236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906183    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-home\") pod \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") "
Sep 08 04:38:25.906236 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906217    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6flb\" (UniqueName: \"kubernetes.io/projected/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-kube-api-access-m6flb\") pod \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") "
Sep 08 04:38:25.906346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-model-cache\") pod \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") "
Sep 08 04:38:25.906346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-dshm\") pod \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") "
Sep 08 04:38:25.906346 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906319    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tls-certs\") pod \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\" (UID: \"5a41ec6f-c903-4e2b-a7bd-649838af3c4f\") "
Sep 08 04:38:25.906492 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906415    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-home" (OuterVolumeSpecName: "home") pod "5a41ec6f-c903-4e2b-a7bd-649838af3c4f" (UID: "5a41ec6f-c903-4e2b-a7bd-649838af3c4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:25.906492 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906431    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a41ec6f-c903-4e2b-a7bd-649838af3c4f" (UID: "5a41ec6f-c903-4e2b-a7bd-649838af3c4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:25.906594 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906512    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tmp-dir\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:25.906594 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906530    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-home\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:25.906657 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.906611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a41ec6f-c903-4e2b-a7bd-649838af3c4f" (UID: "5a41ec6f-c903-4e2b-a7bd-649838af3c4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:25.908269 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.908247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a41ec6f-c903-4e2b-a7bd-649838af3c4f" (UID: "5a41ec6f-c903-4e2b-a7bd-649838af3c4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:38:25.908269 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.908257    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-kube-api-access-m6flb" (OuterVolumeSpecName: "kube-api-access-m6flb") pod "5a41ec6f-c903-4e2b-a7bd-649838af3c4f" (UID: "5a41ec6f-c903-4e2b-a7bd-649838af3c4f"). InnerVolumeSpecName "kube-api-access-m6flb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:38:25.908381 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:25.908244    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-dshm" (OuterVolumeSpecName: "dshm") pod "5a41ec6f-c903-4e2b-a7bd-649838af3c4f" (UID: "5a41ec6f-c903-4e2b-a7bd-649838af3c4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 04:38:26.007334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.007306    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6flb\" (UniqueName: \"kubernetes.io/projected/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-kube-api-access-m6flb\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:26.007334 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.007331    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-model-cache\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:26.007455 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.007345    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-dshm\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:26.007455 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.007356    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a41ec6f-c903-4e2b-a7bd-649838af3c4f-tls-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:26.158359 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.158307    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ede9e8-f33a-4f06-8b27-5f5f49f7c613" path="/var/lib/kubelet/pods/36ede9e8-f33a-4f06-8b27-5f5f49f7c613/volumes"
Sep 08 04:38:26.197133 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.197111    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"]
Sep 08 04:38:26.200632 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:26.200612    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e4456688-kserve-577c5646cf-hc58z"]
Sep 08 04:38:28.158476 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:28.158446    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a41ec6f-c903-4e2b-a7bd-649838af3c4f" path="/var/lib/kubelet/pods/5a41ec6f-c903-4e2b-a7bd-649838af3c4f/volumes"
Sep 08 04:38:30.068076 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068040    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"]
Sep 08 04:38:30.068423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068315    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a41ec6f-c903-4e2b-a7bd-649838af3c4f" containerName="main"
Sep 08 04:38:30.068423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068327    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a41ec6f-c903-4e2b-a7bd-649838af3c4f" containerName="main"
Sep 08 04:38:30.068423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068337    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ede9e8-f33a-4f06-8b27-5f5f49f7c613" containerName="main"
Sep 08 04:38:30.068423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068342    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ede9e8-f33a-4f06-8b27-5f5f49f7c613" containerName="main"
Sep 08 04:38:30.068423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068403    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ede9e8-f33a-4f06-8b27-5f5f49f7c613" containerName="main"
Sep 08 04:38:30.068423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.068412    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a41ec6f-c903-4e2b-a7bd-649838af3c4f" containerName="main"
Sep 08 04:38:30.071161 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.071144    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.074502 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.074479    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 04:38:30.074650 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.074631    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 04:38:30.074722 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.074686    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 04:38:30.074722 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.074694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npqmj\""
Sep 08 04:38:30.074722 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.074710    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 04:38:30.080448 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.080422    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"]
Sep 08 04:38:30.233713 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.233691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-metrics-certs\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.233851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.233728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c9353e-98c4-434e-8c9e-c9abe935762a-openshift-service-ca\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.233851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.233798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcwc\" (UniqueName: \"kubernetes.io/projected/00c9353e-98c4-434e-8c9e-c9abe935762a-kube-api-access-tpcwc\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.233851 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.233825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-maas-api-tls\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.335033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.334982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-metrics-certs\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.335033 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.335018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c9353e-98c4-434e-8c9e-c9abe935762a-openshift-service-ca\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.335174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.335038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcwc\" (UniqueName: \"kubernetes.io/projected/00c9353e-98c4-434e-8c9e-c9abe935762a-kube-api-access-tpcwc\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.335174 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.335071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-maas-api-tls\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.335677 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.335660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c9353e-98c4-434e-8c9e-c9abe935762a-openshift-service-ca\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.337375 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.337357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-metrics-certs\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.337460 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.337443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-maas-api-tls\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.342178 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.342156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcwc\" (UniqueName: \"kubernetes.io/projected/00c9353e-98c4-434e-8c9e-c9abe935762a-kube-api-access-tpcwc\") pod \"maas-api-7df9d55888-m8k72\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.381606 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.381584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:30.649491 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.649466    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"]
Sep 08 04:38:30.651060 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:38:30.651037    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c9353e_98c4_434e_8c9e_c9abe935762a.slice/crio-2347b9183a56cab182439a5892928e626e9375c9086e81182f3284fb34aa8aad WatchSource:0}: Error finding container 2347b9183a56cab182439a5892928e626e9375c9086e81182f3284fb34aa8aad: Status 404 returned error can't find the container with id 2347b9183a56cab182439a5892928e626e9375c9086e81182f3284fb34aa8aad
Sep 08 04:38:30.652293 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.652272    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 04:38:30.904398 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:30.904334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72" event={"ID":"00c9353e-98c4-434e-8c9e-c9abe935762a","Type":"ContainerStarted","Data":"2347b9183a56cab182439a5892928e626e9375c9086e81182f3284fb34aa8aad"}
Sep 08 04:38:31.744053 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:31.744018    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:38:31.910445 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:31.910411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72" event={"ID":"00c9353e-98c4-434e-8c9e-c9abe935762a","Type":"ContainerStarted","Data":"d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a"}
Sep 08 04:38:31.910599 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:31.910540    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:31.928626 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:31.928583    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72" podStartSLOduration=1.479993106 podStartE2EDuration="1.928569356s" totalImagesPullingTime="448.57625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:38:31.927437954 +0000 UTC m=+2292.285917513" watchObservedRunningTime="2026-09-08 04:38:31.928569356 +0000 UTC m=+2292.287048914"
Sep 08 04:38:37.703289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.703255    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"]
Sep 08 04:38:37.703785 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.703441    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72" podUID="00c9353e-98c4-434e-8c9e-c9abe935762a" containerName="maas-api" containerID="cri-o://d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a" gracePeriod=30
Sep 08 04:38:37.708311 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.708284    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:37.738254 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.738229    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"]
Sep 08 04:38:37.741392 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.741370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.752698 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.752674    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"]
Sep 08 04:38:37.889287 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.889253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b520a1f-5ca9-4fda-97a6-847e34f50666-maas-api-tls\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.889423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.889295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b520a1f-5ca9-4fda-97a6-847e34f50666-metrics-certs\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.889423 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.889395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b520a1f-5ca9-4fda-97a6-847e34f50666-openshift-service-ca\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.889496 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.889437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmz4\" (UniqueName: \"kubernetes.io/projected/9b520a1f-5ca9-4fda-97a6-847e34f50666-kube-api-access-9qmz4\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.931634 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.931604    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:37.936188 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.936165    2583 generic.go:356] "Generic (PLEG): container finished" podID="00c9353e-98c4-434e-8c9e-c9abe935762a" containerID="d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a" exitCode=0
Sep 08 04:38:37.936289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.936203    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"
Sep 08 04:38:37.936289 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.936250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72" event={"ID":"00c9353e-98c4-434e-8c9e-c9abe935762a","Type":"ContainerDied","Data":"d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a"}
Sep 08 04:38:37.936366 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.936297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-m8k72" event={"ID":"00c9353e-98c4-434e-8c9e-c9abe935762a","Type":"ContainerDied","Data":"2347b9183a56cab182439a5892928e626e9375c9086e81182f3284fb34aa8aad"}
Sep 08 04:38:37.936366 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.936319    2583 scope.go:117] "RemoveContainer" containerID="d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a"
Sep 08 04:38:37.943828 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.943812    2583 scope.go:117] "RemoveContainer" containerID="d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a"
Sep 08 04:38:37.944081 ip-10-0-130-111 kubenswrapper[2583]: E0908 04:38:37.944062    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a\": container with ID starting with d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a not found: ID does not exist" containerID="d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a"
Sep 08 04:38:37.944137 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.944089    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a"} err="failed to get container status \"d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a\": rpc error: code = NotFound desc = could not find container \"d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a\": container with ID starting with d12e1bacf30d84f2e1ae948e02fffce02e570467b4e08d68e4917fcad1886b2a not found: ID does not exist"
Sep 08 04:38:37.989978 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.989926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b520a1f-5ca9-4fda-97a6-847e34f50666-maas-api-tls\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.989978 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.989956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b520a1f-5ca9-4fda-97a6-847e34f50666-metrics-certs\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.990108 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.989999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b520a1f-5ca9-4fda-97a6-847e34f50666-openshift-service-ca\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.990108 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.990025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmz4\" (UniqueName: \"kubernetes.io/projected/9b520a1f-5ca9-4fda-97a6-847e34f50666-kube-api-access-9qmz4\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.990614 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.990595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b520a1f-5ca9-4fda-97a6-847e34f50666-openshift-service-ca\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.992217 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.992197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b520a1f-5ca9-4fda-97a6-847e34f50666-maas-api-tls\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.992308 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.992248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b520a1f-5ca9-4fda-97a6-847e34f50666-metrics-certs\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:37.998594 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:37.998575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmz4\" (UniqueName: \"kubernetes.io/projected/9b520a1f-5ca9-4fda-97a6-847e34f50666-kube-api-access-9qmz4\") pod \"maas-api-cf5b5fd48-j6bq9\" (UID: \"9b520a1f-5ca9-4fda-97a6-847e34f50666\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:38.090365 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.090341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-maas-api-tls\") pod \"00c9353e-98c4-434e-8c9e-c9abe935762a\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") "
Sep 08 04:38:38.090463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.090391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c9353e-98c4-434e-8c9e-c9abe935762a-openshift-service-ca\") pod \"00c9353e-98c4-434e-8c9e-c9abe935762a\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") "
Sep 08 04:38:38.090463 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.090416    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpcwc\" (UniqueName: \"kubernetes.io/projected/00c9353e-98c4-434e-8c9e-c9abe935762a-kube-api-access-tpcwc\") pod \"00c9353e-98c4-434e-8c9e-c9abe935762a\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") "
Sep 08 04:38:38.090574 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.090503    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-metrics-certs\") pod \"00c9353e-98c4-434e-8c9e-c9abe935762a\" (UID: \"00c9353e-98c4-434e-8c9e-c9abe935762a\") "
Sep 08 04:38:38.090908 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.090885    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c9353e-98c4-434e-8c9e-c9abe935762a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00c9353e-98c4-434e-8c9e-c9abe935762a" (UID: "00c9353e-98c4-434e-8c9e-c9abe935762a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 04:38:38.092085 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.092062    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c9353e-98c4-434e-8c9e-c9abe935762a-kube-api-access-tpcwc" (OuterVolumeSpecName: "kube-api-access-tpcwc") pod "00c9353e-98c4-434e-8c9e-c9abe935762a" (UID: "00c9353e-98c4-434e-8c9e-c9abe935762a"). InnerVolumeSpecName "kube-api-access-tpcwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 04:38:38.092177 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.092147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00c9353e-98c4-434e-8c9e-c9abe935762a" (UID: "00c9353e-98c4-434e-8c9e-c9abe935762a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:38:38.092225 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.092217    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00c9353e-98c4-434e-8c9e-c9abe935762a" (UID: "00c9353e-98c4-434e-8c9e-c9abe935762a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 04:38:38.096365 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.096338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:38.192087 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.192061    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-metrics-certs\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:38.192087 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.192084    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c9353e-98c4-434e-8c9e-c9abe935762a-maas-api-tls\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:38.192226 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.192093    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c9353e-98c4-434e-8c9e-c9abe935762a-openshift-service-ca\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:38.192226 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.192102    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcwc\" (UniqueName: \"kubernetes.io/projected/00c9353e-98c4-434e-8c9e-c9abe935762a-kube-api-access-tpcwc\") on node \"ip-10-0-130-111.ec2.internal\" DevicePath \"\""
Sep 08 04:38:38.244594 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.244517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:38:38.255501 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.255418    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"]
Sep 08 04:38:38.257105 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.257085    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-m8k72"]
Sep 08 04:38:38.362154 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.362133    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"]
Sep 08 04:38:38.365019 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:38:38.364993    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b520a1f_5ca9_4fda_97a6_847e34f50666.slice/crio-e01b0fb1152b04f5c6d758a0f7cac45f8c45a1cf42d0dd4d3b8fa35e4553c901 WatchSource:0}: Error finding container e01b0fb1152b04f5c6d758a0f7cac45f8c45a1cf42d0dd4d3b8fa35e4553c901: Status 404 returned error can't find the container with id e01b0fb1152b04f5c6d758a0f7cac45f8c45a1cf42d0dd4d3b8fa35e4553c901
Sep 08 04:38:38.942719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.942659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9" event={"ID":"9b520a1f-5ca9-4fda-97a6-847e34f50666","Type":"ContainerStarted","Data":"5cf7b5fdc3ba48bd5bc07cd864ecbf4fda2364963ed082c6da4c34ff3c5658d8"}
Sep 08 04:38:38.942719 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.942714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9" event={"ID":"9b520a1f-5ca9-4fda-97a6-847e34f50666","Type":"ContainerStarted","Data":"e01b0fb1152b04f5c6d758a0f7cac45f8c45a1cf42d0dd4d3b8fa35e4553c901"}
Sep 08 04:38:38.943054 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.942776    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:38.960978 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:38.960937    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9" podStartSLOduration=1.676623982 podStartE2EDuration="1.960923242s" totalImagesPullingTime="284.29926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:38:38.959634944 +0000 UTC m=+2299.318114515" watchObservedRunningTime="2026-09-08 04:38:38.960923242 +0000 UTC m=+2299.319402801"
Sep 08 04:38:40.158808 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:40.158776    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c9353e-98c4-434e-8c9e-c9abe935762a" path="/var/lib/kubelet/pods/00c9353e-98c4-434e-8c9e-c9abe935762a/volumes"
Sep 08 04:38:44.951475 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:44.951444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-j6bq9"
Sep 08 04:38:46.345958 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:46.345923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:38:53.548229 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:38:53.548196    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:39:02.245110 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:02.245070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dccpl"]
Sep 08 04:39:04.727517 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:04.727476    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-574c4d4cdb-ttn42_703da780-afd1-4e25-a0ed-e9946fb4d4b8/authorino/0.log"
Sep 08 04:39:11.041192 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:11.041163    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-5prhf_9aad681d-b91c-440c-85dc-4f326e591c93/copy-manifests/0.log"
Sep 08 04:39:11.047793 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:11.047770    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-5prhf_9aad681d-b91c-440c-85dc-4f326e591c93/manager/0.log"
Sep 08 04:39:11.815786 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:11.815758    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-cf5b5fd48-j6bq9_9b520a1f-5ca9-4fda-97a6-847e34f50666/maas-api/0.log"
Sep 08 04:39:13.039060 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:13.038997    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-574c4d4cdb-ttn42_703da780-afd1-4e25-a0ed-e9946fb4d4b8/authorino/0.log"
Sep 08 04:39:13.401867 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:13.401807    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gkn4k_84a71f0a-2d06-4b91-ac86-44754b674b43/kuadrant-console-plugin/0.log"
Sep 08 04:39:13.655986 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:13.655929    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-dccpl_957ca297-c9b6-4c20-8ed3-149731a44ee4/limitador/0.log"
Sep 08 04:39:14.537723 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:14.537701    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5859ff6677-lllzs_6dbf819d-bfb6-4491-80bd-60238fb4bf27/kube-auth-proxy/0.log"
Sep 08 04:39:15.998794 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:15.998769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jgl67_0ab44fd5-a0fb-4fe5-b9f9-6311204f8a05/main/0.log"
Sep 08 04:39:16.134344 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:16.134318    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cqnnh_774282f2-e389-46a1-9001-71537cb4c482/main/0.log"
Sep 08 04:39:20.552381 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:20.552352    2583 scope.go:117] "RemoveContainer" containerID="fff8d1b17536483084c6b2dcfe0946b8f646c9a3bb7330841ac0c0bc2f9015d7"
Sep 08 04:39:24.938566 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:24.938535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lrg6q_b0ab2194-4d54-4200-91fa-6d67c62cec94/global-pull-secret-syncer/0.log"
Sep 08 04:39:25.090462 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:25.090429    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-btsdv_f955d2b4-001f-4502-9c12-8fb652166bee/konnectivity-agent/0.log"
Sep 08 04:39:25.111861 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:25.111839    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-111.ec2.internal_bb5647e1cfe7a0d9b5f7192d30c19c7d/haproxy/0.log"
Sep 08 04:39:29.611212 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:29.611177    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-574c4d4cdb-ttn42_703da780-afd1-4e25-a0ed-e9946fb4d4b8/authorino/0.log"
Sep 08 04:39:29.708059 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:29.708035    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gkn4k_84a71f0a-2d06-4b91-ac86-44754b674b43/kuadrant-console-plugin/0.log"
Sep 08 04:39:29.779180 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:29.779155    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-dccpl_957ca297-c9b6-4c20-8ed3-149731a44ee4/limitador/0.log"
Sep 08 04:39:31.760466 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:31.760439    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqbgr_4292fef7-0a29-4cab-9bdf-a64b1d477cc6/node-exporter/0.log"
Sep 08 04:39:31.794592 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:31.794565    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqbgr_4292fef7-0a29-4cab-9bdf-a64b1d477cc6/kube-rbac-proxy/0.log"
Sep 08 04:39:31.823934 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:31.823915    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqbgr_4292fef7-0a29-4cab-9bdf-a64b1d477cc6/init-textfile/0.log"
Sep 08 04:39:34.491039 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.491003    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/2.log"
Sep 08 04:39:34.498993 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.498968    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rqfd_6931a442-2287-43dd-b61f-93ffa6e89c5b/console-operator/3.log"
Sep 08 04:39:34.828147 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.828112    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"]
Sep 08 04:39:34.828459 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.828442    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c9353e-98c4-434e-8c9e-c9abe935762a" containerName="maas-api"
Sep 08 04:39:34.828554 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.828462    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c9353e-98c4-434e-8c9e-c9abe935762a" containerName="maas-api"
Sep 08 04:39:34.828611 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.828556    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c9353e-98c4-434e-8c9e-c9abe935762a" containerName="maas-api"
Sep 08 04:39:34.831702 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.831681    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.834040 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.834010    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cchdf\"/\"default-dockercfg-lk299\""
Sep 08 04:39:34.834889 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.834873    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cchdf\"/\"kube-root-ca.crt\""
Sep 08 04:39:34.834982 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.834877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cchdf\"/\"openshift-service-ca.crt\""
Sep 08 04:39:34.839038 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.839016    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"]
Sep 08 04:39:34.877728 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.877705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngptf\" (UniqueName: \"kubernetes.io/projected/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-kube-api-access-ngptf\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.877852 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.877735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-sys\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.877920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.877844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-proc\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.877920 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.877878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-podres\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.878013 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.877921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-lib-modules\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979149 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-proc\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979265 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-podres\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979265 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-lib-modules\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979265 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngptf\" (UniqueName: \"kubernetes.io/projected/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-kube-api-access-ngptf\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979265 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-proc\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979527 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-podres\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979527 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-sys\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979527 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979338    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-lib-modules\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.979527 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.979339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-sys\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:34.987895 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:34.987876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngptf\" (UniqueName: \"kubernetes.io/projected/ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f-kube-api-access-ngptf\") pod \"perf-node-gather-daemonset-fx952\" (UID: \"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:35.142175 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:35.142125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:35.409516 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:35.409448    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"]
Sep 08 04:39:35.412363 ip-10-0-130-111 kubenswrapper[2583]: W0908 04:39:35.412342    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce083a30_96e4_4fe4_9b8e_4f66c9cfbc2f.slice/crio-6ff5665a10e48c02fa07b894199ae471182f978099050976656b279831659679 WatchSource:0}: Error finding container 6ff5665a10e48c02fa07b894199ae471182f978099050976656b279831659679: Status 404 returned error can't find the container with id 6ff5665a10e48c02fa07b894199ae471182f978099050976656b279831659679
Sep 08 04:39:35.777454 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:35.777427    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-r2vsj_2e181c82-660a-47df-b138-a3b81c374946/volume-data-source-validator/0.log"
Sep 08 04:39:36.172581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:36.172509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952" event={"ID":"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f","Type":"ContainerStarted","Data":"2f100a81a2844aaef91c4d65c0a95bbe7aeb935d596ab78e0ce49cd8ced81587"}
Sep 08 04:39:36.172581 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:36.172578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952" event={"ID":"ce083a30-96e4-4fe4-9b8e-4f66c9cfbc2f","Type":"ContainerStarted","Data":"6ff5665a10e48c02fa07b894199ae471182f978099050976656b279831659679"}
Sep 08 04:39:36.172775 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:36.172714    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:36.188679 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:36.188621    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952" podStartSLOduration=2.188607825 podStartE2EDuration="2.188607825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:39:36.188270975 +0000 UTC m=+2356.546750533" watchObservedRunningTime="2026-09-08 04:39:36.188607825 +0000 UTC m=+2356.547087385"
Sep 08 04:39:36.968078 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:36.968045    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5h4q_4a6e4a1f-44cd-45a0-ba63-82faba49d733/dns/0.log"
Sep 08 04:39:36.988729 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:36.988701    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5h4q_4a6e4a1f-44cd-45a0-ba63-82faba49d733/kube-rbac-proxy/0.log"
Sep 08 04:39:37.079607 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:37.079585    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rjjg2_02b6f1af-7eab-47aa-a84d-66a045c2dbdf/dns-node-resolver/0.log"
Sep 08 04:39:37.699609 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:37.699579    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xjmjj_6af1c09d-0ba6-4998-891f-0c61ed0f0f35/node-ca/0.log"
Sep 08 04:39:39.034293 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:39.034258    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5859ff6677-lllzs_6dbf819d-bfb6-4491-80bd-60238fb4bf27/kube-auth-proxy/0.log"
Sep 08 04:39:40.047498 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:40.047470    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gh8zx_bcb0eb0f-c7aa-4858-b2bb-d779782a7ad0/serve-healthcheck-canary/0.log"
Sep 08 04:39:40.534207 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:40.534162    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-jpbbn_6f092e55-100b-4423-b95f-be552520d428/insights-operator/0.log"
Sep 08 04:39:40.535369 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:40.535344    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-jpbbn_6f092e55-100b-4423-b95f-be552520d428/insights-operator/1.log"
Sep 08 04:39:40.555615 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:40.555596    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xb48_bb3e4f96-a892-4829-98c3-9284063b83fa/kube-rbac-proxy/0.log"
Sep 08 04:39:40.581265 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:40.581247    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xb48_bb3e4f96-a892-4829-98c3-9284063b83fa/exporter/0.log"
Sep 08 04:39:40.603162 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:40.603143    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xb48_bb3e4f96-a892-4829-98c3-9284063b83fa/extractor/0.log"
Sep 08 04:39:42.185656 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:42.185628    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-fx952"
Sep 08 04:39:43.182604 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:43.182557    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-5prhf_9aad681d-b91c-440c-85dc-4f326e591c93/manager/0.log"
Sep 08 04:39:43.204349 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:43.204325    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-5prhf_9aad681d-b91c-440c-85dc-4f326e591c93/copy-manifests/0.log"
Sep 08 04:39:44.830176 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:44.830144    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58d7df9668-7jwnc_afd765a2-c2c6-4919-bc0d-eff1b3e114ab/manager/0.log"
Sep 08 04:39:45.476036 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:45.476008    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5859ff6677-lllzs_6dbf819d-bfb6-4491-80bd-60238fb4bf27/kube-auth-proxy/0.log"
Sep 08 04:39:47.455848 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:47.455816    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5859ff6677-lllzs_6dbf819d-bfb6-4491-80bd-60238fb4bf27/kube-auth-proxy/0.log"
Sep 08 04:39:50.039082 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:50.039049    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-hbj79_52c19623-5420-4d23-bf2f-64c8df25d568/migrator/0.log"
Sep 08 04:39:50.059313 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:50.059288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-hbj79_52c19623-5420-4d23-bf2f-64c8df25d568/graceful-termination/0.log"
Sep 08 04:39:51.352253 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.352210    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9nwc7_1609f9a3-09f2-487e-adc0-ed2436741052/kube-multus/0.log"
Sep 08 04:39:51.531872 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.531853    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/kube-multus-additional-cni-plugins/0.log"
Sep 08 04:39:51.554540 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.554517    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/egress-router-binary-copy/0.log"
Sep 08 04:39:51.573015 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.572997    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/cni-plugins/0.log"
Sep 08 04:39:51.595051 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.595032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/bond-cni-plugin/0.log"
Sep 08 04:39:51.623141 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.623090    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/routeoverride-cni/0.log"
Sep 08 04:39:51.646608 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.646591    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/whereabouts-cni-bincopy/0.log"
Sep 08 04:39:51.669281 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.669265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gts66_6825e490-6fdf-4a0a-ae3f-2bbe9ea09fc5/whereabouts-cni/0.log"
Sep 08 04:39:51.965205 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.965153    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kj9xx_4d77fff3-3663-446a-af43-cfd608056894/network-metrics-daemon/0.log"
Sep 08 04:39:51.982300 ip-10-0-130-111 kubenswrapper[2583]: I0908 04:39:51.982280    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kj9xx_4d77fff3-3663-446a-af43-cfd608056894/kube-rbac-proxy/0.log"