Sep 11 13:50:48.676039 ip-10-0-128-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 13:50:48.676052 ip-10-0-128-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 13:50:48.676062 ip-10-0-128-105 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 13:50:48.676379 ip-10-0-128-105 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 13:50:58.913339 ip-10-0-128-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 13:50:58.913360 ip-10-0-128-105 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cd47682e57c44f8fbaa9073d1d4dca1d --
Sep 11 13:52:30.317302 ip-10-0-128-105 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 13:52:30.768399 ip-10-0-128-105 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:52:30.768399 ip-10-0-128-105 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 13:52:30.768399 ip-10-0-128-105 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:52:30.768399 ip-10-0-128-105 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 13:52:30.768399 ip-10-0-128-105 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:52:30.771266 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.771178    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 13:52:30.775690 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775667    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:52:30.775690 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775688    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:52:30.775690 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775692    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:52:30.775690 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775695    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:52:30.775690 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775698    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775701    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775705    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775710    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775713    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775716    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775719    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775721    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775724    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775727    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775730    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775732    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775736    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775738    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775741    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775744    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775746    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775749    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775752    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775754    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:52:30.775872 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775757    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775759    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775764    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775767    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775770    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775773    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775779    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775782    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775785    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775788    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775791    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775794    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775797    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775799    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775803    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775806    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775808    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775811    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775814    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:52:30.776364 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775816    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775819    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775822    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775824    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775827    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775829    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775834    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775837    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775841    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775843    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775846    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775849    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775852    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775854    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775857    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775859    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775862    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775865    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775867    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:52:30.776838 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775870    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775872    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775875    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775877    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775880    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775883    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775886    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775888    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775893    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775896    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775899    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775901    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775904    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775907    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775911    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775913    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775916    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775919    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775921    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775936    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:52:30.777316 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775939    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775942    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775945    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775947    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775950    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775953    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775955    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775958    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775961    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775963    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775966    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775968    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775971    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775973    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775976    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775979    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:52:30.777799 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.775981    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:52:30.778549 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778536    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:52:30.778549 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778548    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778552    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778555    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778558    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778560    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778563    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778567    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778570    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778572    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778575    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778577    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778580    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778583    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778585    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778588    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778590    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778593    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778596    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778599    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778601    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:52:30.778602 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778604    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778606    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778609    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778612    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778615    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778617    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778620    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778623    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778625    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778628    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778630    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778632    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778635    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778638    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778640    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778645    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778648    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778651    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778654    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778657    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:52:30.779140 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778659    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778662    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778664    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778667    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778670    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778672    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778675    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778677    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778679    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778683    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778686    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778688    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778691    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778694    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778698    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778701    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778704    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778706    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778709    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778712    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:52:30.779622 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778715    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778718    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778720    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778723    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778725    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778728    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778732    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778736    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778739    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778742    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778744    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778748    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778752    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778755    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778758    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778761    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778764    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778766    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778769    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778772    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:52:30.780141 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778774    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778777    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778780    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778783    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778785    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778788    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778791    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778795    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778799    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778802    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778804    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778807    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778810    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778812    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778815    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778817    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778820    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.778823    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779222    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:52:30.780635 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779227    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779230    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779233    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779236    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779238    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779241    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779244    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779247    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779250    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779253    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779255    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779258    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779261    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779264    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779266    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779269    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779272    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779275    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779283    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:52:30.781128 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779286    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779289    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779291    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779294    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779297    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779300    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779302    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779305    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779308    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779310    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779313    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779316    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779318    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779321    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779323    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779326    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779330    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779333    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779336    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:52:30.781587 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779339    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779342    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779344    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779347    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779350    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779353    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779356    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779358    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779361    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779363    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779367    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779369    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779372    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779375    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779378    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779380    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779384    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779387    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779391    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:52:30.782073 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779393    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779396    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779398    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779401    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779406    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779408    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779411    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779413    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779416    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779418    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779421    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779423    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779426    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779428    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779431    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779433    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779435    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779438    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779440    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779443    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:52:30.782522 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779445    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779449    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779452    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779458    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779461    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779464    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779467    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779470    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779472    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779475    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779477    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779480    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779482    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779485    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779487    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779489    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779492    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779494    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779497    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779499    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:52:30.783018 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.779501    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779588    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779636    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779644    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779649    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779653    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779656    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779660    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779665    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779669    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779672    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779675    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779678    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779681    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779684    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779687    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779690    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779694    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779698    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779702    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779705    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779708    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779711    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779714    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 13:52:30.783500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779718    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779721    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779724    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779728    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779731    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779734    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779737    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779740    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779743    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779747    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779750    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779753    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779756    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779759    2561 flags.go:64] FLAG: --enable-server="true"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779762    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779767    2561 flags.go:64] FLAG: --event-burst="100"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779770    2561 flags.go:64] FLAG: --event-qps="50"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779773    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779775    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779781    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779785    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779801    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779805    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779809    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779812    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 13:52:30.784120 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779815    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779819    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779822    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779825    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779828    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779831    2561 flags.go:64] FLAG: --feature-gates=""
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779835    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779838    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779841    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779844    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779847    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779850    2561 flags.go:64] FLAG: --help="false"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779853    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-128-105.ec2.internal"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779856    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779859    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779861    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779865    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779868    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779871    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779874    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779877    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779880    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779883    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779886    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 13:52:30.784878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779889    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779892    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779896    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779899    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779902    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779905    2561 flags.go:64] FLAG: --lock-file=""
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779908    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779911    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779914    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779920    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779934    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779938    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779941    2561 flags.go:64] FLAG: --logging-format="text"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779944    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779947    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779950    2561 flags.go:64] FLAG: --manifest-url=""
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779953    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779957    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779960    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779965    2561 flags.go:64] FLAG: --max-pods="110"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779968    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779970    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779973    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779976    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779980    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 13:52:30.785657 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779983    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779986    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779992    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779996    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.779999    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780002    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780004    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780009    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780012    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780015    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780020    2561 flags.go:64] FLAG: --port="10250"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780023    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780025    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cae19339f28d3326"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780029    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780032    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780035    2561 flags.go:64] FLAG: --register-node="true"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780038    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780042    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780050    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780053    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780056    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780060    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780063    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780066    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780069    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 13:52:30.786332 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780072    2561 flags.go:64] FLAG: --runonce="false"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780075    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780078    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780081    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780084    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780087    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780090    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780093    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780096    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780099    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780102    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780116    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780121    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780125    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780128    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780131    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780136    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780140    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780143    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780150    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780157    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780160    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780163    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780166    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780169    2561 flags.go:64] FLAG: --v="2"
Sep 11 13:52:30.786957 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780173    2561 flags.go:64] FLAG: --version="false"
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780178    2561 flags.go:64] FLAG: --vmodule=""
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780182    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.780185    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780343    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780347    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780350    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780354    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780357    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780360    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780363    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780366    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780369    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780372    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780375    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780378    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780381    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780384    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780387    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780389    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780392    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:52:30.787571 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780394    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780397    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780401    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780406    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780409    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780412    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780416    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780419    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780422    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780425    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780427    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780430    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780433    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780441    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780444    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780447    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780449    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780452    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780455    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:52:30.788181 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780457    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780460    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780462    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780465    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780467    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780470    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780472    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780475    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780477    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780480    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780482    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780485    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780487    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780490    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780493    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780496    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780500    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780503    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780507    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780511    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:52:30.788660 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780513    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780516    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780518    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780521    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780524    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780526    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780529    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780540    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780543    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780545    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780548    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780551    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780553    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780556    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780558    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780561    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780563    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780566    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:52:30.789188 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780568    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780571    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780573    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780581    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780584    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780588    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780591    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780594    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780597    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780601    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780604    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780606    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780610    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780612    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780615    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780618    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780620    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780623    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780625    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780628    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:52:30.789653 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780630    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780639    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780642    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.780644    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.781296    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.789375    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.789394    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.789415    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.789421    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789476    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789481    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789485    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789488    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789490    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789493    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:52:30.790217 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789497    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789501    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789505    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789508    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789511    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789513    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789516    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789519    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789522    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789524    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789527    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789530    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789533    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789535    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789538    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789541    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789543    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789546    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789549    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:52:30.790596 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789552    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789554    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789557    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789560    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789564    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789566    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789569    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789572    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789574    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789577    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789580    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789583    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789585    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789587    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789590    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789592    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789595    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789598    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789601    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789603    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:52:30.791085 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789606    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789608    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789611    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789613    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789616    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789618    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789621    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789623    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789626    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789629    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789631    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789634    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789638    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789642    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789644    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789647    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789650    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789654    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789658    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:52:30.791610 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789662    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789665    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789668    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789671    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789673    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789676    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789679    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789681    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789684    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789686    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789689    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789691    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789694    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789696    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789699    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789702    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789704    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789707    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789709    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789712    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:52:30.792090 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789714    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789717    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789719    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789722    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789724    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789727    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789730    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789733    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789735    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789738    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789741    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789744    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789746    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789749    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789751    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:52:30.792593 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.789756    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789880    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789885    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789888    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789890    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789893    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789896    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789898    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789901    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789903    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789906    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789909    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789912    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789914    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789916    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789919    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789921    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789940    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789945    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789948    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:52:30.792982 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789951    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789954    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789956    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789959    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789962    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789964    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789967    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789970    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789973    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789977    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789979    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789982    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789984    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789987    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789990    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789993    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789995    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.789998    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790000    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790003    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:52:30.793460 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790005    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790008    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790011    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790013    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790016    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790018    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790021    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790023    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790026    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790028    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790031    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790035    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790039    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790041    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790044    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790046    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790049    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790052    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790055    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:52:30.794008 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790058    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790060    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790065    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790068    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790071    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790074    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790077    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790080    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790083    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790085    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790088    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790090    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790093    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790095    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790098    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790100    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790103    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790106    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790108    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:52:30.794462 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790111    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790113    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790116    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790118    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790121    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790123    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790126    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790129    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790131    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790134    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790136    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790139    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790142    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790144    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790147    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790150    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790152    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790158    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790161    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790163    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:52:30.794938 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790168    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:30.790171    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.790176    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.790290    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.792636    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.793772    2561 server.go:1019] "Starting client certificate rotation"
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.793870    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:52:30.795422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.793909    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:52:30.796625 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.796611    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:52:30.800702 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.800678    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:52:30.815280 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.815259    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 11 13:52:30.818756 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.818741    2561 log.go:25] "Validated CRI v1 image API"
Sep 11 13:52:30.819557 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.819537    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 13:52:30.822810 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.822789    2561 fs.go:135] Filesystem UUIDs: map[6dfdbcc6-32b7-4352-ac74-29fe52a5ae2a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d56ba69f-17bb-4aae-a09d-45919714e2c5:/dev/nvme0n1p3]
Sep 11 13:52:30.822876 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.822808    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 13:52:30.828531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.828420    2561 manager.go:216] Machine: {Timestamp:2026-09-11 13:52:30.827070876 +0000 UTC m=+0.379641438 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101084 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a41ffc76b9c08193a761028c503f9 SystemUUID:ec2a41ff-c76b-9c08-193a-761028c503f9 BootID:cd47682e-57c4-4f8f-baa9-073d1d4dca1d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:8d:16:59:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:8d:16:59:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:bf:e5:b0:63:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 13:52:30.828531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.828524    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 13:52:30.828662 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.828605    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 13:52:30.831430 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.831402    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 13:52:30.831634 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.831434    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 13:52:30.832382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.832371    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 13:52:30.832423 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.832385    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 13:52:30.832423 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.832413    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:52:30.832478 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.832427    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:52:30.833755 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.833743    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 13:52:30.833978 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.833961    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:52:30.838341 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838328    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:52:30.838458 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838448    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 13:52:30.838491 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838479    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 13:52:30.838522 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838491    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 13:52:30.838522 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838504    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 13:52:30.838522 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838514    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 11 13:52:30.838522 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838523    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 13:52:30.838974 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838961    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:52:30.839027 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.838985    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:52:30.841472 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.841456    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 13:52:30.846904 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.846885    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 13:52:30.847001 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.846919    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 13:52:30.849192 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849173    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849197    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849208    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849216    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849225    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849231    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849245    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849250    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849257    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849263    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849271    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 13:52:30.849271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.849277    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 13:52:30.850167 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.850157    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 13:52:30.850167 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.850167    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 13:52:30.851176 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.851144    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 13:52:30.851263 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.851207    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 13:52:30.853822 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.853805    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 13:52:30.854299 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.854286    2561 server.go:1268] "Started kubelet"
Sep 11 13:52:30.854441 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.854416    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 13:52:30.854504 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.854408    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 13:52:30.854504 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.854492    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 13:52:30.855161 ip-10-0-128-105 systemd[1]: Started Kubernetes Kubelet.
Sep 11 13:52:30.857165 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.856141    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 13:52:30.857165 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.856363    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 13:52:30.861505 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.860246    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 13:52:30.864425 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.864407    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 13:52:30.864507 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.863338    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d448939ff98da3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.854253987 +0000 UTC m=+0.406824545,LastTimestamp:2026-09-11 13:52:30.854253987 +0000 UTC m=+0.406824545,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.865015 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.864998    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 13:52:30.865124 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.865105    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:30.865785 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.865768    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 13:52:30.865785 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.865781    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 13:52:30.865940 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.865801    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 13:52:30.865940 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.865913    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 13:52:30.865940 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.865921    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 13:52:30.866100 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866085    2561 factory.go:55] Registering systemd factory
Sep 11 13:52:30.866130 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866103    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 11 13:52:30.866316 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866305    2561 factory.go:153] Registering CRI-O factory
Sep 11 13:52:30.866368 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866319    2561 factory.go:223] Registration of the crio container factory successfully
Sep 11 13:52:30.866413 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866370    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 13:52:30.866413 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866389    2561 factory.go:103] Registering Raw factory
Sep 11 13:52:30.866413 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.866402    2561 manager.go:1196] Started watching for new ooms in manager
Sep 11 13:52:30.867470 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.867454    2561 manager.go:349] Starting recovery of all containers
Sep 11 13:52:30.870495 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.870465    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 13:52:30.870608 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.870588    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 13:52:30.879562 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.879544    2561 manager.go:354] Recovery completed
Sep 11 13:52:30.884565 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.884548    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:30.886482 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.886463    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:30.886572 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.886500    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:30.886572 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.886516    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:30.887116 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887102    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 13:52:30.887166 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887116    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:30.887166 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887116    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 13:52:30.887166 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887154    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:52:30.887835 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887820    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:30.887893 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887847    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:30.887893 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.887861    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:30.888671 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.888592    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a1e5549f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.886483103 +0000 UTC m=+0.439053667,LastTimestamp:2026-09-11 13:52:30.886483103 +0000 UTC m=+0.439053667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.889239 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.889226    2561 policy_none.go:49] "None policy: Start"
Sep 11 13:52:30.889299 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.889242    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 13:52:30.889299 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.889252    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 13:52:30.891127 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.891110    2561 policy_none.go:47] "Start"
Sep 11 13:52:30.898467 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.898401    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a1e5b20f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-105.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.886507023 +0000 UTC m=+0.439077585,LastTimestamp:2026-09-11 13:52:30.886507023 +0000 UTC m=+0.439077585,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.910041 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.909961    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a1e5ee37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-105.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.886522423 +0000 UTC m=+0.439092984,LastTimestamp:2026-09-11 13:52:30.886522423 +0000 UTC m=+0.439092984,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.920554    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-105.ec2.internal.18d44893a1e5549f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a1e5549f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.886483103 +0000 UTC m=+0.439053667,LastTimestamp:2026-09-11 13:52:30.887834574 +0000 UTC m=+0.440405133,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.924213    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.924683    2561 manager.go:337] "Starting Device Plugin manager"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.924724    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.924736    2561 server.go:87] "Starting device plugin registration server"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925149    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925164    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925412    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925508    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925516    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925631    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925675    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.925698    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.925744    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.926124    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.926182    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.927614    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q8hlj"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.931726    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.931754    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-105.ec2.internal.18d44893a1e5b20f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a1e5b20f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-105.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.886507023 +0000 UTC m=+0.439077585,LastTimestamp:2026-09-11 13:52:30.887854014 +0000 UTC m=+0.440424574,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:30.935993    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q8hlj"
Sep 11 13:52:30.936879 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.936292    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-105.ec2.internal.18d44893a1e5ee37\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a1e5ee37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-105.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.886522423 +0000 UTC m=+0.439092984,LastTimestamp:2026-09-11 13:52:30.887865789 +0000 UTC m=+0.440436348,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:30.943962 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:30.943890    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-105.ec2.internal.18d44893a4ce56e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-105.ec2.internal,UID:ip-10-0-128-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-105.ec2.internal,},FirstTimestamp:2026-09-11 13:52:30.935308006 +0000 UTC m=+0.487878552,LastTimestamp:2026-09-11 13:52:30.935308006 +0000 UTC m=+0.487878552,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-105.ec2.internal,}"
Sep 11 13:52:31.026318 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.026182    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:31.026318 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.026202    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"]
Sep 11 13:52:31.026318 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.026291    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:31.027266 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027249    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:31.027366 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027277    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:31.027366 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027287    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:31.027366 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027249    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:31.027366 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027334    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:31.027366 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027345    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:31.027366 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.027363    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.029622 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.029609    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:31.029743 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.029726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.029798 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.029762    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:31.030301 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.030287    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:31.030382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.030306    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:31.030382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.030315    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:31.030382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.030344    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:31.030382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.030362    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:31.030382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.030370    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:31.032505 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.032487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.032609 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.032513    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:52:31.033213 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.033193    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:52:31.033302 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.033226    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:52:31.033302 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.033242    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:52:31.043356 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.043339    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.043430 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.043358    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-105.ec2.internal\": node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:31.055943 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.055916    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-105.ec2.internal\" not found" node="ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.060264 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.060245    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-105.ec2.internal\" not found" node="ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.074989 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.074971    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:31.167195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.167157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a58f6efd61682cc40d607bb0cdab47d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal\" (UID: \"a58f6efd61682cc40d607bb0cdab47d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.167195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.167197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58f6efd61682cc40d607bb0cdab47d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal\" (UID: \"a58f6efd61682cc40d607bb0cdab47d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.167405 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.167220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/694fbd47ec3c35fbb6cd78edfdf87922-config\") pod \"kube-apiserver-proxy-ip-10-0-128-105.ec2.internal\" (UID: \"694fbd47ec3c35fbb6cd78edfdf87922\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.175267 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.175242    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:31.268213 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.268160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a58f6efd61682cc40d607bb0cdab47d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal\" (UID: \"a58f6efd61682cc40d607bb0cdab47d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.268213 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.268214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58f6efd61682cc40d607bb0cdab47d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal\" (UID: \"a58f6efd61682cc40d607bb0cdab47d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.268410 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.268282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a58f6efd61682cc40d607bb0cdab47d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal\" (UID: \"a58f6efd61682cc40d607bb0cdab47d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.268410 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.268298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58f6efd61682cc40d607bb0cdab47d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal\" (UID: \"a58f6efd61682cc40d607bb0cdab47d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.268410 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.268349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/694fbd47ec3c35fbb6cd78edfdf87922-config\") pod \"kube-apiserver-proxy-ip-10-0-128-105.ec2.internal\" (UID: \"694fbd47ec3c35fbb6cd78edfdf87922\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.268410 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.268379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/694fbd47ec3c35fbb6cd78edfdf87922-config\") pod \"kube-apiserver-proxy-ip-10-0-128-105.ec2.internal\" (UID: \"694fbd47ec3c35fbb6cd78edfdf87922\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.276270 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.276247    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:31.359804 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.359742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.363945 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:31.363911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:31.377226 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:31.377203    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:32.662826 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:32.662792    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:32.763270 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:32.763236    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:32.864004 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:32.863935    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:32.964440 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:32.964409    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-105.ec2.internal\" not found"
Sep 11 13:52:32.978585 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:32.978561    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 13:52:33.009311 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.009288    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:52:33.020959 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.020940    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:52:33.024272 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.024257    2561 apiserver.go:52] "Watching apiserver"
Sep 11 13:52:33.035557 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.035537    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 13:52:33.035865 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.035847    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fp4jl","openshift-image-registry/node-ca-g4phg","openshift-multus/multus-additional-cni-plugins-cv947","openshift-multus/multus-cwpvn","openshift-multus/network-metrics-daemon-rfb6q","openshift-network-diagnostics/network-check-target-vx7x6","openshift-network-operator/iptables-alerter-wc2nl","kube-system/konnectivity-agent-8t86n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm","openshift-cluster-node-tuning-operator/tuned-kffkf","openshift-dns/node-resolver-7h9r7","openshift-ovn-kubernetes/ovnkube-node-z96fx"]
Sep 11 13:52:33.038348 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.038306    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.038446 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.038404    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:33.041316 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.041299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.043829 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.043814    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x9jjt\""
Sep 11 13:52:33.044006 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.043989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.044499 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.044481    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.044773 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.044756    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.045323 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.045310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 13:52:33.046558 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.046545    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.046605 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.046572    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 13:52:33.046687 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.046674    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.046864 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.046845    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.046958 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.046882    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 13:52:33.047109 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.047093    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 13:52:33.047109 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.047099    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-47k4j\""
Sep 11 13:52:33.049244 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.049229    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.049335 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.049285    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:33.049662 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.049648    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 13:52:33.050060 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.050039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 13:52:33.050162 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.050147    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:33.051061 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.051045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dqwrz\""
Sep 11 13:52:33.052110 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.052089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:33.052182 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.052166    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:33.054661 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.054644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.056556 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/513fe987-6474-4973-b14e-14020d75bdd8-kubelet-config\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.056633 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.056633 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-cnibin\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056633 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-conf-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056735 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-etc-kubernetes\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056735 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752sr\" (UniqueName: \"kubernetes.io/projected/c5373e77-d971-4400-b35d-cd4bbe543ae5-kube-api-access-752sr\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.056735 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-cni-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056735 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-multus-certs\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056735 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4xd\" (UniqueName: \"kubernetes.io/projected/70628814-f5ff-4599-a973-035afd41d226-kube-api-access-td4xd\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-cni-binary-copy\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-kubelet\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-netns\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-cni-multus\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-system-cni-dir\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-cnibin\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.056892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nw5t\" (UniqueName: \"kubernetes.io/projected/6dc62400-a4d7-4692-bd1a-24c878e56351-kube-api-access-7nw5t\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.056974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-system-cni-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/513fe987-6474-4973-b14e-14020d75bdd8-dbus\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-socket-dir-parent\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09075216-8345-474b-b0ee-063064283279-multus-daemon-config\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxbx\" (UniqueName: \"kubernetes.io/projected/09075216-8345-474b-b0ee-063064283279-kube-api-access-5zxbx\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-k8s-cni-cncf-io\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057136 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-cni-bin\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-hostroot\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5373e77-d971-4400-b35d-cd4bbe543ae5-serviceca\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5373e77-d971-4400-b35d-cd4bbe543ae5-host\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09075216-8345-474b-b0ee-063064283279-cni-binary-copy\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.057541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.057331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.059347 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.059312    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.059434 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.059360    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.059970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.059956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.060594 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.060577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 13:52:33.060796 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.060777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6622v\""
Sep 11 13:52:33.061256 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.061238    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w9dsn\""
Sep 11 13:52:33.062688 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.062669    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 13:52:33.062824 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.062807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.064065 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.064044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-smcc8\""
Sep 11 13:52:33.064065 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.064056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.064357 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.064342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 13:52:33.064941 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.064920    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:52:33.065519 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.065500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.065966 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.065949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.066214 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.066112    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.067149 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.067133    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zsp4t\""
Sep 11 13:52:33.067244 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.067173    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.068162 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.068146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.068923 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.068898    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.069541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.069527    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4nczq\""
Sep 11 13:52:33.069614 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.069534    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.069943 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.069929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 13:52:33.072068 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.072056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 13:52:33.073222 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.073207    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 13:52:33.074712 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.074693    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 13:52:33.074819 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.074727    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 13:52:33.074819 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.074740    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hj8l8\""
Sep 11 13:52:33.074819 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.074747    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 13:52:33.074819 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.074789    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 13:52:33.078829 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.078812    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal"]
Sep 11 13:52:33.079392 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.079380    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:52:33.079468 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.079430    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"
Sep 11 13:52:33.081468 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.081450    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:52:33.091545 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.091529    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6s9c6"
Sep 11 13:52:33.098865 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.098846    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:52:33.099329 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.099309    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal"]
Sep 11 13:52:33.104736 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.104717    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6s9c6"
Sep 11 13:52:33.126798 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.126755    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:47:32 +0000 UTC" deadline="2028-04-26 03:48:18.820424923 +0000 UTC"
Sep 11 13:52:33.126798 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.126793    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14221h55m45.693633772s"
Sep 11 13:52:33.151667 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.151645    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 13:52:33.158402 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-cni-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158507 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-multus-certs\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158507 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.158507 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fgwb\" (UniqueName: \"kubernetes.io/projected/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-kube-api-access-7fgwb\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.158507 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-kubelet\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158507 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-modprobe-d\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d45bb637-dafd-45e8-95dd-3731ba76f452-hosts-file\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-multus-certs\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f73fcb4-d524-42e1-855e-b2c63d237f8d-host-slash\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-cni-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-kubelet\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/707077db-11b9-4e88-9a4b-23b441e474fd-konnectivity-ca\") pod \"konnectivity-agent-8t86n\" (UID: \"707077db-11b9-4e88-9a4b-23b441e474fd\") " pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-cni-bin\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovnkube-script-lib\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-systemd\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-netns\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-cni-multus\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.158700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-system-cni-dir\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-cnibin\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-netns\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nw5t\" (UniqueName: \"kubernetes.io/projected/6dc62400-a4d7-4692-bd1a-24c878e56351-kube-api-access-7nw5t\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-cni-multus\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-node-log\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-system-cni-dir\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-cnibin\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-systemd-units\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-slash\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-ovn\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-log-socket\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-cni-netd\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-hostroot\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.158997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-kubelet\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-env-overrides\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-hostroot\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5373e77-d971-4400-b35d-cd4bbe543ae5-serviceca\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-etc-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovnkube-config\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysconfig\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-tmp\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f73fcb4-d524-42e1-855e-b2c63d237f8d-iptables-alerter-script\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-systemd\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-cnibin\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-conf-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-socket-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-cnibin\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-conf-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45bb637-dafd-45e8-95dd-3731ba76f452-tmp-dir\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8fz4\" (UniqueName: \"kubernetes.io/projected/4f73fcb4-d524-42e1-855e-b2c63d237f8d-kube-api-access-s8fz4\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752sr\" (UniqueName: \"kubernetes.io/projected/c5373e77-d971-4400-b35d-cd4bbe543ae5-kube-api-access-752sr\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.159970 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4xd\" (UniqueName: \"kubernetes.io/projected/70628814-f5ff-4599-a973-035afd41d226-kube-api-access-td4xd\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-lib-modules\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/707077db-11b9-4e88-9a4b-23b441e474fd-agent-certs\") pod \"konnectivity-agent-8t86n\" (UID: \"707077db-11b9-4e88-9a4b-23b441e474fd\") " pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-cni-binary-copy\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5373e77-d971-4400-b35d-cd4bbe543ae5-serviceca\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2mv\" (UniqueName: \"kubernetes.io/projected/da5ddc0b-7a20-4aae-9003-49fb9f60661b-kube-api-access-2g2mv\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-tuned\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-system-cni-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxbx\" (UniqueName: \"kubernetes.io/projected/09075216-8345-474b-b0ee-063064283279-kube-api-access-5zxbx\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-device-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-var-lib-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/513fe987-6474-4973-b14e-14020d75bdd8-dbus\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-system-cni-dir\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-socket-dir-parent\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.160768 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09075216-8345-474b-b0ee-063064283279-multus-daemon-config\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqwq\" (UniqueName: \"kubernetes.io/projected/780f1dab-47f8-4550-86b5-3d262fa658cd-kube-api-access-wtqwq\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.159972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/513fe987-6474-4973-b14e-14020d75bdd8-dbus\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc62400-a4d7-4692-bd1a-24c878e56351-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysctl-d\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-multus-socket-dir-parent\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-sys\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-var-lib-kubelet\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-k8s-cni-cncf-io\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-cni-bin\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-sys-fs\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysctl-conf\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gsk4\" (UniqueName: \"kubernetes.io/projected/d45bb637-dafd-45e8-95dd-3731ba76f452-kube-api-access-8gsk4\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-host\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.161387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-var-lib-cni-bin\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-run-netns\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.160453    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5373e77-d971-4400-b35d-cd4bbe543ae5-host\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-host-run-k8s-cni-cncf-io\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.160516    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:33.6604961 +0000 UTC m=+2.028085540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5373e77-d971-4400-b35d-cd4bbe543ae5-host\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc62400-a4d7-4692-bd1a-24c878e56351-cni-binary-copy\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09075216-8345-474b-b0ee-063064283279-cni-binary-copy\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09075216-8345-474b-b0ee-063064283279-multus-daemon-config\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-kubernetes\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-run\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovn-node-metrics-cert\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/513fe987-6474-4973-b14e-14020d75bdd8-kubelet-config\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.162282 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/513fe987-6474-4973-b14e-14020d75bdd8-kubelet-config\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-etc-kubernetes\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09075216-8345-474b-b0ee-063064283279-etc-kubernetes\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.160931    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-registration-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.160948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09075216-8345-474b-b0ee-063064283279-cni-binary-copy\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.163147 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.160967    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:33.66095612 +0000 UTC m=+2.028545547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:33.174404 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.173569    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:52:33.174404 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.173614    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:52:33.174404 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.173638    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:33.174404 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.173726    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:33.673698917 +0000 UTC m=+2.041288361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:33.174404 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.174134    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 13:52:33.178817 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.178657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxbx\" (UniqueName: \"kubernetes.io/projected/09075216-8345-474b-b0ee-063064283279-kube-api-access-5zxbx\") pod \"multus-cwpvn\" (UID: \"09075216-8345-474b-b0ee-063064283279\") " pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.178959 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.178693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752sr\" (UniqueName: \"kubernetes.io/projected/c5373e77-d971-4400-b35d-cd4bbe543ae5-kube-api-access-752sr\") pod \"node-ca-g4phg\" (UID: \"c5373e77-d971-4400-b35d-cd4bbe543ae5\") " pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.178959 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.178698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nw5t\" (UniqueName: \"kubernetes.io/projected/6dc62400-a4d7-4692-bd1a-24c878e56351-kube-api-access-7nw5t\") pod \"multus-additional-cni-plugins-cv947\" (UID: \"6dc62400-a4d7-4692-bd1a-24c878e56351\") " pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.178959 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.178882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4xd\" (UniqueName: \"kubernetes.io/projected/70628814-f5ff-4599-a973-035afd41d226-kube-api-access-td4xd\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.224240 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.224200    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694fbd47ec3c35fbb6cd78edfdf87922.slice/crio-3368d0cbdf9e94c434d9c1f821f0e7852deb807ee004b9bc5e8f0e1184e9c479 WatchSource:0}: Error finding container 3368d0cbdf9e94c434d9c1f821f0e7852deb807ee004b9bc5e8f0e1184e9c479: Status 404 returned error can't find the container with id 3368d0cbdf9e94c434d9c1f821f0e7852deb807ee004b9bc5e8f0e1184e9c479
Sep 11 13:52:33.224442 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.224416    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58f6efd61682cc40d607bb0cdab47d3.slice/crio-317a31c42bc24ae57a6daf02f488f4da9d8c40b8448c043586913a90600a3061 WatchSource:0}: Error finding container 317a31c42bc24ae57a6daf02f488f4da9d8c40b8448c043586913a90600a3061: Status 404 returned error can't find the container with id 317a31c42bc24ae57a6daf02f488f4da9d8c40b8448c043586913a90600a3061
Sep 11 13:52:33.228134 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.228114    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:52:33.262010 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.261984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-etc-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262010 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovnkube-config\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysconfig\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-tmp\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f73fcb4-d524-42e1-855e-b2c63d237f8d-iptables-alerter-script\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-systemd\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-etc-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysconfig\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-socket-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-systemd\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45bb637-dafd-45e8-95dd-3731ba76f452-tmp-dir\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.262230 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8fz4\" (UniqueName: \"kubernetes.io/projected/4f73fcb4-d524-42e1-855e-b2c63d237f8d-kube-api-access-s8fz4\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-socket-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-lib-modules\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/707077db-11b9-4e88-9a4b-23b441e474fd-agent-certs\") pod \"konnectivity-agent-8t86n\" (UID: \"707077db-11b9-4e88-9a4b-23b441e474fd\") " pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2mv\" (UniqueName: \"kubernetes.io/projected/da5ddc0b-7a20-4aae-9003-49fb9f60661b-kube-api-access-2g2mv\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-tuned\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-device-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-var-lib-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqwq\" (UniqueName: \"kubernetes.io/projected/780f1dab-47f8-4550-86b5-3d262fa658cd-kube-api-access-wtqwq\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysctl-d\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-sys\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-device-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-var-lib-kubelet\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d45bb637-dafd-45e8-95dd-3731ba76f452-tmp-dir\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-sys-fs\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysctl-conf\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.262786 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gsk4\" (UniqueName: \"kubernetes.io/projected/d45bb637-dafd-45e8-95dd-3731ba76f452-kube-api-access-8gsk4\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-host\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-var-lib-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-run-netns\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-var-lib-kubelet\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovnkube-config\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-sys-fs\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f73fcb4-d524-42e1-855e-b2c63d237f8d-iptables-alerter-script\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-lib-modules\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-host\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysctl-d\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-run-netns\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-sys\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-sysctl-conf\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-kubernetes\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-kubernetes\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.263576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-run\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-openvswitch\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.262976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovn-node-metrics-cert\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-run\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-registration-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fgwb\" (UniqueName: \"kubernetes.io/projected/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-kube-api-access-7fgwb\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-modprobe-d\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-registration-dir\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-modprobe-d\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/780f1dab-47f8-4550-86b5-3d262fa658cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d45bb637-dafd-45e8-95dd-3731ba76f452-hosts-file\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f73fcb4-d524-42e1-855e-b2c63d237f8d-host-slash\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/707077db-11b9-4e88-9a4b-23b441e474fd-konnectivity-ca\") pod \"konnectivity-agent-8t86n\" (UID: \"707077db-11b9-4e88-9a4b-23b441e474fd\") " pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d45bb637-dafd-45e8-95dd-3731ba76f452-hosts-file\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.264303 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-cni-bin\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f73fcb4-d524-42e1-855e-b2c63d237f8d-host-slash\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovnkube-script-lib\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-cni-bin\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-systemd\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-node-log\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-systemd-units\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-slash\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-ovn\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-systemd\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-log-socket\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-systemd-units\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-cni-netd\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-node-log\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-kubelet\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-run-ovn\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-env-overrides\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-slash\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.264821 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-cni-netd\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-log-socket\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da5ddc0b-7a20-4aae-9003-49fb9f60661b-host-kubelet\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.263798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/707077db-11b9-4e88-9a4b-23b441e474fd-konnectivity-ca\") pod \"konnectivity-agent-8t86n\" (UID: \"707077db-11b9-4e88-9a4b-23b441e474fd\") " pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.264012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovnkube-script-lib\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.264046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da5ddc0b-7a20-4aae-9003-49fb9f60661b-env-overrides\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.264890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/707077db-11b9-4e88-9a4b-23b441e474fd-agent-certs\") pod \"konnectivity-agent-8t86n\" (UID: \"707077db-11b9-4e88-9a4b-23b441e474fd\") " pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.265041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-tmp\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.265333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da5ddc0b-7a20-4aae-9003-49fb9f60661b-ovn-node-metrics-cert\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.265484 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.265346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-etc-tuned\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.282682 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.282646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2mv\" (UniqueName: \"kubernetes.io/projected/da5ddc0b-7a20-4aae-9003-49fb9f60661b-kube-api-access-2g2mv\") pod \"ovnkube-node-z96fx\" (UID: \"da5ddc0b-7a20-4aae-9003-49fb9f60661b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.283132 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.283115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8fz4\" (UniqueName: \"kubernetes.io/projected/4f73fcb4-d524-42e1-855e-b2c63d237f8d-kube-api-access-s8fz4\") pod \"iptables-alerter-wc2nl\" (UID: \"4f73fcb4-d524-42e1-855e-b2c63d237f8d\") " pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.285491 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.285465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqwq\" (UniqueName: \"kubernetes.io/projected/780f1dab-47f8-4550-86b5-3d262fa658cd-kube-api-access-wtqwq\") pod \"aws-ebs-csi-driver-node-6ntmm\" (UID: \"780f1dab-47f8-4550-86b5-3d262fa658cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.285836 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.285814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gsk4\" (UniqueName: \"kubernetes.io/projected/d45bb637-dafd-45e8-95dd-3731ba76f452-kube-api-access-8gsk4\") pod \"node-resolver-7h9r7\" (UID: \"d45bb637-dafd-45e8-95dd-3731ba76f452\") " pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.285904 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.285832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fgwb\" (UniqueName: \"kubernetes.io/projected/c1bb4b6b-7ed7-4244-bccc-90d9864e4992-kube-api-access-7fgwb\") pod \"tuned-kffkf\" (UID: \"c1bb4b6b-7ed7-4244-bccc-90d9864e4992\") " pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.365124 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.365084    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g4phg"
Sep 11 13:52:33.369651 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.369543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cv947"
Sep 11 13:52:33.377584 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.377556    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5373e77_d971_4400_b35d_cd4bbe543ae5.slice/crio-f31d3d622dc689c85b3212fde9e5f53b6fbe467f0e6253ecab6ee62a77dc5b4d WatchSource:0}: Error finding container f31d3d622dc689c85b3212fde9e5f53b6fbe467f0e6253ecab6ee62a77dc5b4d: Status 404 returned error can't find the container with id f31d3d622dc689c85b3212fde9e5f53b6fbe467f0e6253ecab6ee62a77dc5b4d
Sep 11 13:52:33.377993 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.377975    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc62400_a4d7_4692_bd1a_24c878e56351.slice/crio-2ffee29e85b25b53f13e0fba5ee46cfe1b84978c85561b3c96f6726b37afc9bf WatchSource:0}: Error finding container 2ffee29e85b25b53f13e0fba5ee46cfe1b84978c85561b3c96f6726b37afc9bf: Status 404 returned error can't find the container with id 2ffee29e85b25b53f13e0fba5ee46cfe1b84978c85561b3c96f6726b37afc9bf
Sep 11 13:52:33.400390 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.400365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cwpvn"
Sep 11 13:52:33.407272 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.407254    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wc2nl"
Sep 11 13:52:33.409198 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.409168    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09075216_8345_474b_b0ee_063064283279.slice/crio-44996d7b441a09217c75cadf74a615bc7e5c5b19bebd8525ded26857709c117f WatchSource:0}: Error finding container 44996d7b441a09217c75cadf74a615bc7e5c5b19bebd8525ded26857709c117f: Status 404 returned error can't find the container with id 44996d7b441a09217c75cadf74a615bc7e5c5b19bebd8525ded26857709c117f
Sep 11 13:52:33.415514 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.415495    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f73fcb4_d524_42e1_855e_b2c63d237f8d.slice/crio-b7ec8c0cc60b57bf6064043490dc305d0a5e77b3bd32640d7100fe972afe6e32 WatchSource:0}: Error finding container b7ec8c0cc60b57bf6064043490dc305d0a5e77b3bd32640d7100fe972afe6e32: Status 404 returned error can't find the container with id b7ec8c0cc60b57bf6064043490dc305d0a5e77b3bd32640d7100fe972afe6e32
Sep 11 13:52:33.432354 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.432332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:33.438008 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.437990    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm"
Sep 11 13:52:33.441277 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.441254    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707077db_11b9_4e88_9a4b_23b441e474fd.slice/crio-c4c91cf0a132eeede366b34b39d5f8ab5e5b5ed7056d83da4a11f12be0d9aafa WatchSource:0}: Error finding container c4c91cf0a132eeede366b34b39d5f8ab5e5b5ed7056d83da4a11f12be0d9aafa: Status 404 returned error can't find the container with id c4c91cf0a132eeede366b34b39d5f8ab5e5b5ed7056d83da4a11f12be0d9aafa
Sep 11 13:52:33.447201 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.447180    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780f1dab_47f8_4550_86b5_3d262fa658cd.slice/crio-c111d0d099fad84ad1bff3c076b1cc13ba7f473fdd12f0c3727536e70c192cc8 WatchSource:0}: Error finding container c111d0d099fad84ad1bff3c076b1cc13ba7f473fdd12f0c3727536e70c192cc8: Status 404 returned error can't find the container with id c111d0d099fad84ad1bff3c076b1cc13ba7f473fdd12f0c3727536e70c192cc8
Sep 11 13:52:33.461198 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.461176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kffkf"
Sep 11 13:52:33.469409 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.469390    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bb4b6b_7ed7_4244_bccc_90d9864e4992.slice/crio-5e853852dfe6654cf85e4b72c7490dd564aa25460f1b2c38361773a014b90513 WatchSource:0}: Error finding container 5e853852dfe6654cf85e4b72c7490dd564aa25460f1b2c38361773a014b90513: Status 404 returned error can't find the container with id 5e853852dfe6654cf85e4b72c7490dd564aa25460f1b2c38361773a014b90513
Sep 11 13:52:33.479374 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.479358    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7h9r7"
Sep 11 13:52:33.488479 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.488457    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45bb637_dafd_45e8_95dd_3731ba76f452.slice/crio-12a8adee4a233210902ee3e80a9259ab72fa93de4bc1376f3d4d577f006a268f WatchSource:0}: Error finding container 12a8adee4a233210902ee3e80a9259ab72fa93de4bc1376f3d4d577f006a268f: Status 404 returned error can't find the container with id 12a8adee4a233210902ee3e80a9259ab72fa93de4bc1376f3d4d577f006a268f
Sep 11 13:52:33.489112 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.489090    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:33.496746 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:52:33.496722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5ddc0b_7a20_4aae_9003_49fb9f60661b.slice/crio-0d6abd3b4fa11bd3461947c9f41f960827c8e250f98180cae77716c6daab3118 WatchSource:0}: Error finding container 0d6abd3b4fa11bd3461947c9f41f960827c8e250f98180cae77716c6daab3118: Status 404 returned error can't find the container with id 0d6abd3b4fa11bd3461947c9f41f960827c8e250f98180cae77716c6daab3118
Sep 11 13:52:33.627894 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.627806    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:52:33.666045 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.666021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:33.666172 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.666055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:33.666172 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.666155    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:33.666241 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.666174    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:33.666241 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.666197    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:34.666185948 +0000 UTC m=+3.033775368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:33.666241 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.666227    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:34.666209687 +0000 UTC m=+3.033799119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:33.767487 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:33.767449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:33.767659 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.767604    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:52:33.767659 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.767627    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:52:33.767659 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.767638    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:33.767823 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:33.767696    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:34.767678309 +0000 UTC m=+3.135267750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:34.108113 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.107996    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:47:33 +0000 UTC" deadline="2028-04-15 07:37:25.725397046 +0000 UTC"
Sep 11 13:52:34.108113 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.108058    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13961h44m51.617342805s"
Sep 11 13:52:34.113845 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.113817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:34.114005 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.113977    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:34.129553 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.129457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cwpvn" event={"ID":"09075216-8345-474b-b0ee-063064283279","Type":"ContainerStarted","Data":"44996d7b441a09217c75cadf74a615bc7e5c5b19bebd8525ded26857709c117f"}
Sep 11 13:52:34.131820 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.131747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal" event={"ID":"694fbd47ec3c35fbb6cd78edfdf87922","Type":"ContainerStarted","Data":"3368d0cbdf9e94c434d9c1f821f0e7852deb807ee004b9bc5e8f0e1184e9c479"}
Sep 11 13:52:34.137510 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.137314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"0d6abd3b4fa11bd3461947c9f41f960827c8e250f98180cae77716c6daab3118"}
Sep 11 13:52:34.146178 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.146148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kffkf" event={"ID":"c1bb4b6b-7ed7-4244-bccc-90d9864e4992","Type":"ContainerStarted","Data":"5e853852dfe6654cf85e4b72c7490dd564aa25460f1b2c38361773a014b90513"}
Sep 11 13:52:34.157769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.157740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerStarted","Data":"2ffee29e85b25b53f13e0fba5ee46cfe1b84978c85561b3c96f6726b37afc9bf"}
Sep 11 13:52:34.164134 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.164106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g4phg" event={"ID":"c5373e77-d971-4400-b35d-cd4bbe543ae5","Type":"ContainerStarted","Data":"f31d3d622dc689c85b3212fde9e5f53b6fbe467f0e6253ecab6ee62a77dc5b4d"}
Sep 11 13:52:34.171213 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.171148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal" event={"ID":"a58f6efd61682cc40d607bb0cdab47d3","Type":"ContainerStarted","Data":"317a31c42bc24ae57a6daf02f488f4da9d8c40b8448c043586913a90600a3061"}
Sep 11 13:52:34.180928 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.180890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7h9r7" event={"ID":"d45bb637-dafd-45e8-95dd-3731ba76f452","Type":"ContainerStarted","Data":"12a8adee4a233210902ee3e80a9259ab72fa93de4bc1376f3d4d577f006a268f"}
Sep 11 13:52:34.188762 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.188731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm" event={"ID":"780f1dab-47f8-4550-86b5-3d262fa658cd","Type":"ContainerStarted","Data":"c111d0d099fad84ad1bff3c076b1cc13ba7f473fdd12f0c3727536e70c192cc8"}
Sep 11 13:52:34.203836 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.203801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8t86n" event={"ID":"707077db-11b9-4e88-9a4b-23b441e474fd","Type":"ContainerStarted","Data":"c4c91cf0a132eeede366b34b39d5f8ab5e5b5ed7056d83da4a11f12be0d9aafa"}
Sep 11 13:52:34.210722 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.210693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wc2nl" event={"ID":"4f73fcb4-d524-42e1-855e-b2c63d237f8d","Type":"ContainerStarted","Data":"b7ec8c0cc60b57bf6064043490dc305d0a5e77b3bd32640d7100fe972afe6e32"}
Sep 11 13:52:34.676192 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.676153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:34.676397 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.676221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:34.676397 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.676391    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:34.676526 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.676454    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:36.676436081 +0000 UTC m=+5.044025504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:34.676881 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.676861    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:34.676985 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.676932    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:36.67690165 +0000 UTC m=+5.044491079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:34.776700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:34.776666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:34.776951 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.776886    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:52:34.776951 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.776932    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:52:34.776951 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.776946    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:34.777148 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:34.777004    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:36.776987045 +0000 UTC m=+5.144576478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:35.108392 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:35.108299    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:47:33 +0000 UTC" deadline="2028-02-07 14:59:31.165207785 +0000 UTC"
Sep 11 13:52:35.108392 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:35.108346    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12337h6m56.056866792s"
Sep 11 13:52:35.111727 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:35.111702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:35.111859 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:35.111837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:35.111944 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:35.111702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:35.112005 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:35.111961    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:36.112944 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:36.112715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:36.112944 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.112865    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:36.694316 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:36.693722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:36.694316 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:36.693781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:36.694316 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.693924    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:36.694316 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.693975    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:36.694316 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.693993    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:40.693973773 +0000 UTC m=+9.061563209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:36.694316 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.694060    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:40.694043276 +0000 UTC m=+9.061632709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:36.795493 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:36.794849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:36.795493 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.795073    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:52:36.795493 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.795095    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:52:36.795493 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.795108    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:36.795493 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:36.795164    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:40.795146557 +0000 UTC m=+9.162735985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:37.111117 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:37.110973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:37.111299 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:37.111160    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:37.111625 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:37.111595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:37.111744 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:37.111700    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:38.111154 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:38.111122    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:38.111720 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:38.111271    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:39.111676 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:39.111643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:39.112181 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:39.111770    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:39.112181 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:39.111927    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:39.112181 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:39.112008    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:40.111036 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:40.111009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:40.111211 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.111139    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:40.730474 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:40.730432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:40.730892 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:40.730543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:40.730892 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.730664    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:40.730892 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.730719    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:48.730700371 +0000 UTC m=+17.098289806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:40.731125 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.731110    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:40.731177 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.731161    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:48.731145616 +0000 UTC m=+17.098735052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:40.832083 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:40.831396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:40.832083 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.831578    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:52:40.832083 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.831599    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:52:40.832083 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.831613    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:40.832083 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:40.831682    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:52:48.831651924 +0000 UTC m=+17.199241370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:41.111800 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:41.111535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:41.111800 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:41.111588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:41.111800 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:41.111659    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:41.111800 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:41.111711    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:42.114143 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:42.114097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:42.114571 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:42.114224    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:43.111876 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:43.111838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:43.112080 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:43.111887    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:43.112080 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:43.111994    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:43.112195 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:43.112114    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:44.110990 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:44.110961    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:44.111538 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:44.111100    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:45.111608 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:45.111579    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:45.112037 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:45.111699    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:45.112037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:45.111754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:45.112037 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:45.111885    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:46.111229 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:46.111193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:46.111425 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:46.111352    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:47.111745 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:47.111703    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:47.112225 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:47.111748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:47.112225 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:47.111839    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:47.112225 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:47.111952    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:48.111642 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:48.111608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:48.111821 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.111759    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:48.788298 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:48.788258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:48.788499 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:48.788311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:48.788499 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.788427    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:48.788499 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.788432    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:48.788499 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.788487    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:04.788471424 +0000 UTC m=+33.156060850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:52:48.788695 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.788506    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:04.788494867 +0000 UTC m=+33.156084289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:52:48.889185 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:48.889143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:48.889337 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.889298    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:52:48.889337 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.889316    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:52:48.889337 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.889328    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:48.889496 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:48.889390    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:04.889371619 +0000 UTC m=+33.256961055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:52:49.111223 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:49.111136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:49.111384 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:49.111138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:49.111384 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:49.111263    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:49.111384 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:49.111320    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:50.111464 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:50.111429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:50.111981 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:50.111574    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:51.111567 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:51.111531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:51.111567 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:51.111566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:51.112049 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:51.111657    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:51.112049 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:51.111803    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:52.112318 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:52.112290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:52.112683 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:52.112390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:53.111388 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.111080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:53.111551 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.111081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:53.111551 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:53.111498    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:53.111640 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:53.111554    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:53.260558 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.260483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.260809    2561 generic.go:356] "Generic (PLEG): container finished" podID="da5ddc0b-7a20-4aae-9003-49fb9f60661b" containerID="5a4c1fbc26d84f9a4eb33374029f9ade091299d1edf3b6206aa32dfe6f45392d" exitCode=1
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.260875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"ce7ad25ab98aa704f1bd5f178041a7deae8a05f6d5968a868eb25f8e5ff2b241"}
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.260956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"7243bef94ef022fa691951acad7e7b44fd3609f434d48650237647ea272f6ab9"}
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.260990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"cd40d050f1ff812bec996d2746cacc0f6ca4a7f7c1e43fb09a564c57caf0b790"}
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.261021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"f37268fa2c55d6a3563f04447626cdcb00e7181091574d276dcccf24d9976ee8"}
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.261059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerDied","Data":"5a4c1fbc26d84f9a4eb33374029f9ade091299d1edf3b6206aa32dfe6f45392d"}
Sep 11 13:52:53.261203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.261102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"21f94fc14a8d352655fc9f1fd90f5538ca8fdc4a1bcec94ebc5547b0949a1697"}
Sep 11 13:52:53.262295 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.262270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kffkf" event={"ID":"c1bb4b6b-7ed7-4244-bccc-90d9864e4992","Type":"ContainerStarted","Data":"7c0256236d49476e1a3113e1cffbd127bbac5829d5f5512671f69239807a3b60"}
Sep 11 13:52:53.263585 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.263564    2561 generic.go:356] "Generic (PLEG): container finished" podID="6dc62400-a4d7-4692-bd1a-24c878e56351" containerID="325c7d72d41dd7c3e5d681158e98eb225a1e79000c8aabffc4433a9e39bfb24c" exitCode=0
Sep 11 13:52:53.263675 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.263629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerDied","Data":"325c7d72d41dd7c3e5d681158e98eb225a1e79000c8aabffc4433a9e39bfb24c"}
Sep 11 13:52:53.265189 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.265169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g4phg" event={"ID":"c5373e77-d971-4400-b35d-cd4bbe543ae5","Type":"ContainerStarted","Data":"e0b291f4b02a87ef2038a3830ff7b1f351010908834f74c57858569f4478632b"}
Sep 11 13:52:53.266755 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.266738    2561 generic.go:356] "Generic (PLEG): container finished" podID="a58f6efd61682cc40d607bb0cdab47d3" containerID="5ab30698f40e49bc2f18a0fb8cc8919cd52f3408891fbdbdfe100083fc511338" exitCode=0
Sep 11 13:52:53.266844 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.266825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal" event={"ID":"a58f6efd61682cc40d607bb0cdab47d3","Type":"ContainerDied","Data":"5ab30698f40e49bc2f18a0fb8cc8919cd52f3408891fbdbdfe100083fc511338"}
Sep 11 13:52:53.268486 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.268464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7h9r7" event={"ID":"d45bb637-dafd-45e8-95dd-3731ba76f452","Type":"ContainerStarted","Data":"047f85f25262bba13586e3394daf45a5b53fe619e723dc7df4374960163ce252"}
Sep 11 13:52:53.269877 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.269855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm" event={"ID":"780f1dab-47f8-4550-86b5-3d262fa658cd","Type":"ContainerStarted","Data":"efc563d5af3ec8b5d55bd1e49dad09a17cad0cfc05a66377d747b30fa4c014dd"}
Sep 11 13:52:53.271343 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.271327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8t86n" event={"ID":"707077db-11b9-4e88-9a4b-23b441e474fd","Type":"ContainerStarted","Data":"a44aa143a0de57432794f5b6c20f51b5aacbd0e2822083db9a452e313abd97b4"}
Sep 11 13:52:53.272706 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.272682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cwpvn" event={"ID":"09075216-8345-474b-b0ee-063064283279","Type":"ContainerStarted","Data":"467c98d833dee889999fd4ad8fb88225d78422302dbde4d2c29c865096194f78"}
Sep 11 13:52:53.274163 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.274145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal" event={"ID":"694fbd47ec3c35fbb6cd78edfdf87922","Type":"ContainerStarted","Data":"af0ecfe174257933d9a87c03adaa31d10956481b05c8531dc56b5182129fd56e"}
Sep 11 13:52:53.332658 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.332613    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kffkf" podStartSLOduration=2.520910261 podStartE2EDuration="21.332597655s" totalImagesPullingTime="18.811687394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:53.307176148 +0000 UTC m=+21.674765595" watchObservedRunningTime="2026-09-11 13:52:53.332597655 +0000 UTC m=+21.700187098"
Sep 11 13:52:53.332875 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.332855    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-105.ec2.internal" podStartSLOduration=20.332849089 podStartE2EDuration="20.332849089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:53.332453679 +0000 UTC m=+21.700043123" watchObservedRunningTime="2026-09-11 13:52:53.332849089 +0000 UTC m=+21.700438538"
Sep 11 13:52:53.429505 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.429444    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g4phg" podStartSLOduration=2.554686324 podStartE2EDuration="21.429420965s" totalImagesPullingTime="18.874734641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:53.388218806 +0000 UTC m=+21.755808249" watchObservedRunningTime="2026-09-11 13:52:53.429420965 +0000 UTC m=+21.797010409"
Sep 11 13:52:53.486622 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.486571    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7h9r7" podStartSLOduration=2.694749518 podStartE2EDuration="21.486554603s" totalImagesPullingTime="18.791805085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:53.457450414 +0000 UTC m=+21.825039854" watchObservedRunningTime="2026-09-11 13:52:53.486554603 +0000 UTC m=+21.854144046"
Sep 11 13:52:53.541656 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.541615    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cwpvn" podStartSLOduration=2.613574449 podStartE2EDuration="21.541595521s" totalImagesPullingTime="18.928021072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:53.521513982 +0000 UTC m=+21.889103426" watchObservedRunningTime="2026-09-11 13:52:53.541595521 +0000 UTC m=+21.909184964"
Sep 11 13:52:53.541941 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:53.541892    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8t86n" podStartSLOduration=6.653255979 podStartE2EDuration="21.541880642s" totalImagesPullingTime="14.888624663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:53.541521775 +0000 UTC m=+21.909111218" watchObservedRunningTime="2026-09-11 13:52:53.541880642 +0000 UTC m=+21.909470122"
Sep 11 13:52:54.084757 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.084594    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 13:52:54.111271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.111245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:54.111400 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:54.111366    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:54.120294 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.120193    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:52:54.084752559Z","UUID":"93d2029b-f523-4813-a3e5-68f72b569ac0","Handler":null,"Name":"","Endpoint":""}
Sep 11 13:52:54.122117 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.122096    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 13:52:54.122223 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.122133    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 13:52:54.279304 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.279185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal" event={"ID":"a58f6efd61682cc40d607bb0cdab47d3","Type":"ContainerStarted","Data":"aa414f1ffddf2fd9cc824349c120ab64bfdf4ef67b242b8c2d75ed8bb0312e54"}
Sep 11 13:52:54.281402 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.281366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm" event={"ID":"780f1dab-47f8-4550-86b5-3d262fa658cd","Type":"ContainerStarted","Data":"8440a234ce1841ee9dbb9a376301e62dfe92e69b9c6b0c3f642ee0ffa8e59e96"}
Sep 11 13:52:54.283013 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.282969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wc2nl" event={"ID":"4f73fcb4-d524-42e1-855e-b2c63d237f8d","Type":"ContainerStarted","Data":"c2d2c16de3b5cf59f72e549fbb0eb147781c21af130ad79f4a68f20308cf0461"}
Sep 11 13:52:54.327540 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.327492    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wc2nl" podStartSLOduration=7.411659044 podStartE2EDuration="22.327474583s" totalImagesPullingTime="14.915815539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:54.32734723 +0000 UTC m=+22.694936673" watchObservedRunningTime="2026-09-11 13:52:54.327474583 +0000 UTC m=+22.695064025"
Sep 11 13:52:54.327729 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.327583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-105.ec2.internal" podStartSLOduration=21.327574234 podStartE2EDuration="21.327574234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:54.305719448 +0000 UTC m=+22.673308891" watchObservedRunningTime="2026-09-11 13:52:54.327574234 +0000 UTC m=+22.695163750"
Sep 11 13:52:54.714035 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.713828    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:54.714481 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:54.714458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:55.111961 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:55.111866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:55.111961 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:55.111930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:55.112190 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:55.112014    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:55.112190 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:55.112149    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:55.285797 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:55.285732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:55.286289 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:55.286218    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8t86n"
Sep 11 13:52:56.111190 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:56.111156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:56.111483 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:56.111306    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:56.292954 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:56.292927    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 13:52:56.293375 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:56.293336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"2c179542c93ea9348a63ba559075f17abb55a6292f7e9b170a8c326a54ba449d"}
Sep 11 13:52:56.295321 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:56.295297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm" event={"ID":"780f1dab-47f8-4550-86b5-3d262fa658cd","Type":"ContainerStarted","Data":"0f6a5e7d233f6c56a41238e0c59596534c190efd1f7ece24990d6d57b40e9669"}
Sep 11 13:52:56.407964 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:56.407857    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ntmm" podStartSLOduration=3.195061162 podStartE2EDuration="24.407839514s" totalImagesPullingTime="21.212778352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:56.407219743 +0000 UTC m=+24.774809187" watchObservedRunningTime="2026-09-11 13:52:56.407839514 +0000 UTC m=+24.775428956"
Sep 11 13:52:57.111625 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:57.111594    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:57.111778 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:57.111694    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:57.111778 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:57.111747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:57.111878 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:57.111860    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:58.111576 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:58.111540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:52:58.111935 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:58.111664    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:52:58.304709 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:58.304508    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 13:52:58.305098 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:58.305076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"20267cab52957b7dd0b113e6acfe9c0dd800562d7b5b09a442211718d24de27c"}
Sep 11 13:52:58.305317 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:58.305303    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:58.305489 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:58.305474    2561 scope.go:117] "RemoveContainer" containerID="5a4c1fbc26d84f9a4eb33374029f9ade091299d1edf3b6206aa32dfe6f45392d"
Sep 11 13:52:58.325268 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:58.325246    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:59.111448 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.111414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:52:59.111653 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.111414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:52:59.111653 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:59.111515    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:52:59.111653 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:52:59.111600    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:52:59.312902 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.312806    2561 generic.go:356] "Generic (PLEG): container finished" podID="6dc62400-a4d7-4692-bd1a-24c878e56351" containerID="129e39579ad314cd208f9bfa5a9be41b5b37547c768e92708f40a37720578e90" exitCode=0
Sep 11 13:52:59.313087 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.312922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerDied","Data":"129e39579ad314cd208f9bfa5a9be41b5b37547c768e92708f40a37720578e90"}
Sep 11 13:52:59.317552 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.317534    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 13:52:59.317949 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.317929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" event={"ID":"da5ddc0b-7a20-4aae-9003-49fb9f60661b","Type":"ContainerStarted","Data":"850bedc99da0e465e3f3f32035ef661fb21ebee91f4ddfde6f8211beb9176c2d"}
Sep 11 13:52:59.318044 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.318029    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 13:52:59.318249 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.318237    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:59.335731 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.335694    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:52:59.423799 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:52:59.423748    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx" podStartSLOduration=8.61108394 podStartE2EDuration="27.423730858s" totalImagesPullingTime="18.812646918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:59.418067653 +0000 UTC m=+27.785657096" watchObservedRunningTime="2026-09-11 13:52:59.423730858 +0000 UTC m=+27.791320297"
Sep 11 13:53:00.110944 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.110885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:00.111071 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:00.111045    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:53:00.212806 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.212778    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfb6q"]
Sep 11 13:53:00.217667 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.217631    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fp4jl"]
Sep 11 13:53:00.217801 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.217764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:00.217903 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:00.217879    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:53:00.218323 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.218297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vx7x6"]
Sep 11 13:53:00.218542 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.218396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:00.218542 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:00.218488    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:53:00.322671 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.322637    2561 generic.go:356] "Generic (PLEG): container finished" podID="6dc62400-a4d7-4692-bd1a-24c878e56351" containerID="c30fe5aad69a1995c6354dddf7f3cf0e09146366548d3582a61ca28b2e561762" exitCode=0
Sep 11 13:53:00.322823 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.322684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerDied","Data":"c30fe5aad69a1995c6354dddf7f3cf0e09146366548d3582a61ca28b2e561762"}
Sep 11 13:53:00.322823 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.322784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:00.322924 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:00.322892    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:53:00.323055 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:00.323040    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 13:53:01.326009 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:01.325982    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 13:53:02.112050 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:02.112015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:02.112213 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:02.112097    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:53:02.112213 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:02.112118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:02.112213 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:02.112176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:02.112373 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:02.112256    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:53:02.112373 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:02.112319    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:53:02.330355 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:02.330312    2561 generic.go:356] "Generic (PLEG): container finished" podID="6dc62400-a4d7-4692-bd1a-24c878e56351" containerID="022a4b4c6fc11c9a129fa3cc7ce818e5ab3583dbc1da5669640f9a13b747188d" exitCode=0
Sep 11 13:53:02.330759 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:02.330366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerDied","Data":"022a4b4c6fc11c9a129fa3cc7ce818e5ab3583dbc1da5669640f9a13b747188d"}
Sep 11 13:53:02.625663 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:02.625410    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:53:04.111641 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:04.111552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:04.112159 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:04.111552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:04.112159 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.111687    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx7x6" podUID="abcc8642-9a3b-4a54-aa55-ac49439b03f0"
Sep 11 13:53:04.112159 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.111787    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfb6q" podUID="70628814-f5ff-4599-a973-035afd41d226"
Sep 11 13:53:04.112159 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:04.111552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:04.112159 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.111863    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp4jl" podUID="513fe987-6474-4973-b14e-14020d75bdd8"
Sep 11 13:53:04.805895 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:04.805855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:04.806154 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:04.805903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:04.806154 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.806020    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:53:04.806154 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.806073    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret podName:513fe987-6474-4973-b14e-14020d75bdd8 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:36.806057117 +0000 UTC m=+65.173646537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret") pod "global-pull-secret-syncer-fp4jl" (UID: "513fe987-6474-4973-b14e-14020d75bdd8") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:53:04.806154 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.806021    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:53:04.806154 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.806151    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:36.806137827 +0000 UTC m=+65.173727260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:53:04.906317 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:04.906282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:04.906514 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.906420    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:53:04.906514 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.906439    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:53:04.906514 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.906449    2561 projected.go:196] Error preparing data for projected volume kube-api-access-v2vkk for pod openshift-network-diagnostics/network-check-target-vx7x6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:53:04.906514 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:04.906507    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk podName:abcc8642-9a3b-4a54-aa55-ac49439b03f0 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:36.90648689 +0000 UTC m=+65.274076313 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2vkk" (UniqueName: "kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk") pod "network-check-target-vx7x6" (UID: "abcc8642-9a3b-4a54-aa55-ac49439b03f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:53:05.341947 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.341904    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-105.ec2.internal" event="NodeReady"
Sep 11 13:53:05.342317 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.342052    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 13:53:05.438980 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.438946    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-598d5f97dd-wsrs7"]
Sep 11 13:53:05.487333 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.487303    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"]
Sep 11 13:53:05.487485 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.487425    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.502223 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.502196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 13:53:05.502365 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.502274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bjrq5\""
Sep 11 13:53:05.502365 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.502274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 13:53:05.502365 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.502274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 13:53:05.507091 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.507072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"]
Sep 11 13:53:05.507210 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.507197    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.508807 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.508783    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 13:53:05.510456 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.510437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c66zd\""
Sep 11 13:53:05.510568 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.510457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 13:53:05.510779 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.510761    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 13:53:05.511205 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.511190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 13:53:05.517048 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.517026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 13:53:05.527353 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.527310    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fz5x6"]
Sep 11 13:53:05.527481 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.527436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.544543 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.543828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-598d5f97dd-wsrs7"]
Sep 11 13:53:05.544543 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.543872    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"]
Sep 11 13:53:05.544543 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.543992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.550094 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.547977    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 13:53:05.552618 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.552598    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 13:53:05.552735 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.552601    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 13:53:05.553717 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.553697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t9pwf\""
Sep 11 13:53:05.554872 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.554855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 13:53:05.568064 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.567993    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"]
Sep 11 13:53:05.568174 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.568091    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.568174 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.568100    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fz5x6"]
Sep 11 13:53:05.568174 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.568162    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dsn9c"]
Sep 11 13:53:05.572316 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.572296    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 13:53:05.572447 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.572345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 13:53:05.572580 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.572563    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 13:53:05.576144 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.576128    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 13:53:05.588480 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.588428    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"]
Sep 11 13:53:05.588563 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.588505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.588563 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.588523    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"]
Sep 11 13:53:05.588563 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.588542    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dsn9c"]
Sep 11 13:53:05.593016 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.592964    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 13:53:05.593016 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.592970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nllbd\""
Sep 11 13:53:05.593291 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.593276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 13:53:05.611026 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt9n7\" (UniqueName: \"kubernetes.io/projected/616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb-kube-api-access-jt9n7\") pod \"managed-serviceaccount-addon-agent-788d5df8db-7mlgm\" (UID: \"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.611135 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611047    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-image-registry-private-configuration\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611135 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/da7ea9be-9c9f-4379-8380-d1924ca333a3-kube-api-access-d4k24\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.611135 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b618c1b1-c6b9-4751-9ad9-63d160f97e94-ca-trust-extracted\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611277 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-installation-pull-secrets\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611277 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ft28\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-kube-api-access-6ft28\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611277 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-788d5df8db-7mlgm\" (UID: \"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.611277 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da7ea9be-9c9f-4379-8380-d1924ca333a3-tmp\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.611277 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611495 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-bound-sa-token\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611495 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/da7ea9be-9c9f-4379-8380-d1924ca333a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.611495 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-certificates\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.611495 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36bcb94d-4a3b-476a-b64a-8531b0e398f5-cert\") pod \"ingress-canary-fz5x6\" (UID: \"36bcb94d-4a3b-476a-b64a-8531b0e398f5\") " pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.611663 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgtd\" (UniqueName: \"kubernetes.io/projected/36bcb94d-4a3b-476a-b64a-8531b0e398f5-kube-api-access-hmgtd\") pod \"ingress-canary-fz5x6\" (UID: \"36bcb94d-4a3b-476a-b64a-8531b0e398f5\") " pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.611663 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.611561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-trusted-ca\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.637429 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.637382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:05.712285 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-788d5df8db-7mlgm\" (UID: \"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.712455 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da7ea9be-9c9f-4379-8380-d1924ca333a3-tmp\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.712455 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.712455 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.712455 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-bound-sa-token\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.712455 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/da7ea9be-9c9f-4379-8380-d1924ca333a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.712455 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-ca\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:05.712521    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:05.712543    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-certificates\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36bcb94d-4a3b-476a-b64a-8531b0e398f5-cert\") pod \"ingress-canary-fz5x6\" (UID: \"36bcb94d-4a3b-476a-b64a-8531b0e398f5\") " pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgtd\" (UniqueName: \"kubernetes.io/projected/36bcb94d-4a3b-476a-b64a-8531b0e398f5-kube-api-access-hmgtd\") pod \"ingress-canary-fz5x6\" (UID: \"36bcb94d-4a3b-476a-b64a-8531b0e398f5\") " pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d9a6af-284c-4150-926d-f0cd36705cba-config-volume\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da7ea9be-9c9f-4379-8380-d1924ca333a3-tmp\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.712769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjpf\" (UniqueName: \"kubernetes.io/projected/203ef1e2-d1a7-471d-938b-55440446c76e-kube-api-access-cnjpf\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-trusted-ca\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt9n7\" (UniqueName: \"kubernetes.io/projected/616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb-kube-api-access-jt9n7\") pod \"managed-serviceaccount-addon-agent-788d5df8db-7mlgm\" (UID: \"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-image-registry-private-configuration\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/da7ea9be-9c9f-4379-8380-d1924ca333a3-kube-api-access-d4k24\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.712929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:05.712994    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:06.212970402 +0000 UTC m=+34.580559825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b618c1b1-c6b9-4751-9ad9-63d160f97e94-ca-trust-extracted\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-installation-pull-secrets\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ft28\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-kube-api-access-6ft28\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d9a6af-284c-4150-926d-f0cd36705cba-tmp-dir\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.713284 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ef1e2-d1a7-471d-938b-55440446c76e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.713878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-hub\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.713878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-certificates\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.713878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnvd\" (UniqueName: \"kubernetes.io/projected/b9d9a6af-284c-4150-926d-f0cd36705cba-kube-api-access-hcnvd\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.713878 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.713458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b618c1b1-c6b9-4751-9ad9-63d160f97e94-ca-trust-extracted\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.714422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.714399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-trusted-ca\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.717203 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.717160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-image-registry-private-configuration\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.717313 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.717248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-installation-pull-secrets\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.717313 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.717283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/da7ea9be-9c9f-4379-8380-d1924ca333a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.717422 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.717374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-788d5df8db-7mlgm\" (UID: \"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.717884 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.717855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36bcb94d-4a3b-476a-b64a-8531b0e398f5-cert\") pod \"ingress-canary-fz5x6\" (UID: \"36bcb94d-4a3b-476a-b64a-8531b0e398f5\") " pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.737791 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.737756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-bound-sa-token\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.746531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.746502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt9n7\" (UniqueName: \"kubernetes.io/projected/616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb-kube-api-access-jt9n7\") pod \"managed-serviceaccount-addon-agent-788d5df8db-7mlgm\" (UID: \"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.747028 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.747003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4k24\" (UniqueName: \"kubernetes.io/projected/da7ea9be-9c9f-4379-8380-d1924ca333a3-kube-api-access-d4k24\") pod \"klusterlet-addon-workmgr-695dd765dc-5rxbh\" (UID: \"da7ea9be-9c9f-4379-8380-d1924ca333a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.748531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.748498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ft28\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-kube-api-access-6ft28\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:05.749194 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.749161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgtd\" (UniqueName: \"kubernetes.io/projected/36bcb94d-4a3b-476a-b64a-8531b0e398f5-kube-api-access-hmgtd\") pod \"ingress-canary-fz5x6\" (UID: \"36bcb94d-4a3b-476a-b64a-8531b0e398f5\") " pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.811623 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.811578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:05.814552 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d9a6af-284c-4150-926d-f0cd36705cba-config-volume\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.814668 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjpf\" (UniqueName: \"kubernetes.io/projected/203ef1e2-d1a7-471d-938b-55440446c76e-kube-api-access-cnjpf\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.814668 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.814668 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d9a6af-284c-4150-926d-f0cd36705cba-tmp-dir\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.814831 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.814831 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ef1e2-d1a7-471d-938b-55440446c76e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.814831 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-hub\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.814831 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnvd\" (UniqueName: \"kubernetes.io/projected/b9d9a6af-284c-4150-926d-f0cd36705cba-kube-api-access-hcnvd\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.814831 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.815089 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.814834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-ca\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.815145 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.815107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d9a6af-284c-4150-926d-f0cd36705cba-tmp-dir\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.815234 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.815211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d9a6af-284c-4150-926d-f0cd36705cba-config-volume\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.815294 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:05.815220    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:05.815358 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:05.815311    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:53:06.315292223 +0000 UTC m=+34.682881658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:05.815499 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.815480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ef1e2-d1a7-471d-938b-55440446c76e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.817675 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.817649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.817785 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.817681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-ca\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.817785 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.817740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.817897 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.817792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/203ef1e2-d1a7-471d-938b-55440446c76e-hub\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.828640 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.828611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"
Sep 11 13:53:05.835988 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.835958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:05.837373 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.837351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnvd\" (UniqueName: \"kubernetes.io/projected/b9d9a6af-284c-4150-926d-f0cd36705cba-kube-api-access-hcnvd\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:05.838070 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.838050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjpf\" (UniqueName: \"kubernetes.io/projected/203ef1e2-d1a7-471d-938b-55440446c76e-kube-api-access-cnjpf\") pod \"cluster-proxy-proxy-agent-54cd95f7c6-vjkwf\" (UID: \"203ef1e2-d1a7-471d-938b-55440446c76e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:05.856178 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.856155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fz5x6"
Sep 11 13:53:05.877100 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:05.877076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:53:06.112116 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.112029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:06.112291 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.112029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:06.112662 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.112034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:06.116844 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.116822    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:53:06.117001 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.116822    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:53:06.117113 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.117097    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-spk9p\""
Sep 11 13:53:06.117193 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.117144    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k94vs\""
Sep 11 13:53:06.118649 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.118627    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:53:06.118769 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.118658    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:53:06.217960 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.217921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:06.218179 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:06.218156    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:06.218238 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:06.218182    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:06.218290 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:06.218252    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:07.218230895 +0000 UTC m=+35.585820340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:06.318492 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:06.318461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:06.318678 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:06.318578    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:06.318678 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:06.318636    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:53:07.318617978 +0000 UTC m=+35.686207401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:07.226537 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:07.226253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:07.226941 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:07.226726    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:07.226941 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:07.226744    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:07.226941 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:07.226806    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:09.226785395 +0000 UTC m=+37.594374825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:07.327704 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:07.327667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:07.327870 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:07.327826    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:07.327937 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:07.327926    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:53:09.327885235 +0000 UTC m=+37.695474668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:07.432060 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:07.431973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"]
Sep 11 13:53:07.435899 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:53:07.435846    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7ea9be_9c9f_4379_8380_d1924ca333a3.slice/crio-c83570aec5949ed70fb6e0ba87181c8a67cfef07d24fa44385cf58693935b07e WatchSource:0}: Error finding container c83570aec5949ed70fb6e0ba87181c8a67cfef07d24fa44385cf58693935b07e: Status 404 returned error can't find the container with id c83570aec5949ed70fb6e0ba87181c8a67cfef07d24fa44385cf58693935b07e
Sep 11 13:53:07.439564 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:07.439495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fz5x6"]
Sep 11 13:53:07.440362 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:07.440299    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm"]
Sep 11 13:53:07.440867 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:07.440788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"]
Sep 11 13:53:07.443172 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:53:07.443146    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bcb94d_4a3b_476a_b64a_8531b0e398f5.slice/crio-5110cc251e51ff343cd5619407092ff3fdd7b6ceea6c06646da13d35343eddcf WatchSource:0}: Error finding container 5110cc251e51ff343cd5619407092ff3fdd7b6ceea6c06646da13d35343eddcf: Status 404 returned error can't find the container with id 5110cc251e51ff343cd5619407092ff3fdd7b6ceea6c06646da13d35343eddcf
Sep 11 13:53:07.443951 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:53:07.443853    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod203ef1e2_d1a7_471d_938b_55440446c76e.slice/crio-b9c4b49305901ad0b18569c5108cace41eb1f3c45a3d86ef28184012a89bd471 WatchSource:0}: Error finding container b9c4b49305901ad0b18569c5108cace41eb1f3c45a3d86ef28184012a89bd471: Status 404 returned error can't find the container with id b9c4b49305901ad0b18569c5108cace41eb1f3c45a3d86ef28184012a89bd471
Sep 11 13:53:07.444558 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:53:07.444536    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod616fd9c3_7d53_4e33_8ac3_a5f1fc0675fb.slice/crio-eb07b3ac85e83c995ab1b0e1b61576eb7f66ed22df3a6ebe481f5728cfc1172b WatchSource:0}: Error finding container eb07b3ac85e83c995ab1b0e1b61576eb7f66ed22df3a6ebe481f5728cfc1172b: Status 404 returned error can't find the container with id eb07b3ac85e83c995ab1b0e1b61576eb7f66ed22df3a6ebe481f5728cfc1172b
Sep 11 13:53:08.354165 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:08.354126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh" event={"ID":"da7ea9be-9c9f-4379-8380-d1924ca333a3","Type":"ContainerStarted","Data":"c83570aec5949ed70fb6e0ba87181c8a67cfef07d24fa44385cf58693935b07e"}
Sep 11 13:53:08.361577 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:08.360612    2561 generic.go:356] "Generic (PLEG): container finished" podID="6dc62400-a4d7-4692-bd1a-24c878e56351" containerID="50254ce5e7818cdceafaa0dd70bb4c803c86ec07dd8df128d5a00c5086fc2e26" exitCode=0
Sep 11 13:53:08.361577 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:08.360714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerDied","Data":"50254ce5e7818cdceafaa0dd70bb4c803c86ec07dd8df128d5a00c5086fc2e26"}
Sep 11 13:53:08.364132 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:08.364056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" event={"ID":"203ef1e2-d1a7-471d-938b-55440446c76e","Type":"ContainerStarted","Data":"b9c4b49305901ad0b18569c5108cace41eb1f3c45a3d86ef28184012a89bd471"}
Sep 11 13:53:08.370419 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:08.370274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm" event={"ID":"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb","Type":"ContainerStarted","Data":"eb07b3ac85e83c995ab1b0e1b61576eb7f66ed22df3a6ebe481f5728cfc1172b"}
Sep 11 13:53:08.375720 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:08.374438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fz5x6" event={"ID":"36bcb94d-4a3b-476a-b64a-8531b0e398f5","Type":"ContainerStarted","Data":"5110cc251e51ff343cd5619407092ff3fdd7b6ceea6c06646da13d35343eddcf"}
Sep 11 13:53:09.247552 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:09.246946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:09.247552 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:09.247107    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:09.247552 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:09.247123    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:09.247552 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:09.247181    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:13.24716288 +0000 UTC m=+41.614752317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:09.348611 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:09.347710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:09.348611 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:09.348212    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:09.348611 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:09.348275    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:53:13.348257786 +0000 UTC m=+41.715847222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:09.386706 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:09.386639    2561 generic.go:356] "Generic (PLEG): container finished" podID="6dc62400-a4d7-4692-bd1a-24c878e56351" containerID="471b4a46dcac8b242d4b52a2e6f488eb4275ca329adf7791b6eb27a25fe8214a" exitCode=0
Sep 11 13:53:09.387289 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:09.386777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerDied","Data":"471b4a46dcac8b242d4b52a2e6f488eb4275ca329adf7791b6eb27a25fe8214a"}
Sep 11 13:53:10.397051 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:10.397010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cv947" event={"ID":"6dc62400-a4d7-4692-bd1a-24c878e56351","Type":"ContainerStarted","Data":"0ce6a020da35603092c94cafaa2dc44f728d3fd5f1f1bd697b14587a7735874d"}
Sep 11 13:53:10.632447 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:10.632401    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:10.775055 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:10.774955    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:10.893666 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:10.893617    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:13.286672 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:13.286631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:13.287145 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:13.286803    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:13.287145 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:13.286819    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:13.287145 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:13.286876    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:21.286856554 +0000 UTC m=+49.654445980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:13.387588 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:13.387547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:13.387753 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:13.387707    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:13.387807 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:13.387793    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:53:21.387768733 +0000 UTC m=+49.755358174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:16.108599 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:16.108564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:16.208060 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:16.208021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:17.416256 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.416216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh" event={"ID":"da7ea9be-9c9f-4379-8380-d1924ca333a3","Type":"ContainerStarted","Data":"85ce0b14c11cfc31f07fe89e31daf3e31612121ac1a91a26fd4c5761815bec59"}
Sep 11 13:53:17.416709 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.416392    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:17.418005 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.417978    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh"
Sep 11 13:53:17.418609 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.418587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" event={"ID":"203ef1e2-d1a7-471d-938b-55440446c76e","Type":"ContainerStarted","Data":"841ee4dc1100b3bdeff7c78ecb55cc825bbbb050c8118a71f493c7508159f02e"}
Sep 11 13:53:17.418692 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.418623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" event={"ID":"203ef1e2-d1a7-471d-938b-55440446c76e","Type":"ContainerStarted","Data":"7f093a3e7145d3822279ba7776d158e1ffebf522380d1d9f7919e72affa9a3eb"}
Sep 11 13:53:17.418692 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.418637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" event={"ID":"203ef1e2-d1a7-471d-938b-55440446c76e","Type":"ContainerStarted","Data":"961e4e1cf1f2e645a518bbac1c81c707043af07ea2c483b438061a4c78c7f665"}
Sep 11 13:53:17.419966 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.419944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm" event={"ID":"616fd9c3-7d53-4e33-8ac3-a5f1fc0675fb","Type":"ContainerStarted","Data":"f706a401f2767f21ddddd15a5538192a6d102b93109a8d8e9367184f0f33820a"}
Sep 11 13:53:17.421409 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.421387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fz5x6" event={"ID":"36bcb94d-4a3b-476a-b64a-8531b0e398f5","Type":"ContainerStarted","Data":"90b85ff8d868983104c8dc0ca07035ba1485a698579ceb1379fc8c1084f5bcaf"}
Sep 11 13:53:17.448500 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.448460    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695dd765dc-5rxbh" podStartSLOduration=9.513147817 podStartE2EDuration="18.448446903s" totalImagesPullingTime="8.935299086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:17.448213494 +0000 UTC m=+45.815802937" watchObservedRunningTime="2026-09-11 13:53:17.448446903 +0000 UTC m=+45.816036337"
Sep 11 13:53:17.448951 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.448926    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cv947" podStartSLOduration=14.536227299 podStartE2EDuration="45.448897911s" totalImagesPullingTime="30.912670612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:10.430693495 +0000 UTC m=+38.798282932" watchObservedRunningTime="2026-09-11 13:53:17.448897911 +0000 UTC m=+45.816487424"
Sep 11 13:53:17.597603 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.597550    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fz5x6" podStartSLOduration=3.665203935 podStartE2EDuration="12.597530759s" totalImagesPullingTime="8.932326824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:17.550604564 +0000 UTC m=+45.918194006" watchObservedRunningTime="2026-09-11 13:53:17.597530759 +0000 UTC m=+45.965120214"
Sep 11 13:53:17.597998 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.597964    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" podStartSLOduration=9.651562124 podStartE2EDuration="18.597954027s" totalImagesPullingTime="8.946391903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:17.59559842 +0000 UTC m=+45.963187864" watchObservedRunningTime="2026-09-11 13:53:17.597954027 +0000 UTC m=+45.965543478"
Sep 11 13:53:17.640018 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.639965    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-788d5df8db-7mlgm" podStartSLOduration=9.710132359 podStartE2EDuration="18.639949714s" totalImagesPullingTime="8.929817355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:17.639691296 +0000 UTC m=+46.007280741" watchObservedRunningTime="2026-09-11 13:53:17.639949714 +0000 UTC m=+46.007539147"
Sep 11 13:53:17.748614 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.748520    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:17.881344 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:17.881309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:18.598791 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:18.598754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:18.700340 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:18.700281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:19.528276 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:19.528241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log" limit=1024
Sep 11 13:53:21.348177 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:21.348131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:21.348575 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:21.348303    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:21.348575 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:21.348322    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:21.348575 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:21.348417    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:37.348401381 +0000 UTC m=+65.715990807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:21.448526 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:21.448486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:21.448691 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:21.448635    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:21.448731 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:21.448691    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:53:37.448676761 +0000 UTC m=+65.816266182 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:32.636252 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:32.636225    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z96fx"
Sep 11 13:53:36.854543 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.854502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:53:36.855056 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.854549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:36.857494 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.857463    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:53:36.857650 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.857517    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:53:36.864736 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:36.864715    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 13:53:36.864823 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:36.864783    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs podName:70628814-f5ff-4599-a973-035afd41d226 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:40.864760618 +0000 UTC m=+129.232350052 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs") pod "network-metrics-daemon-rfb6q" (UID: "70628814-f5ff-4599-a973-035afd41d226") : secret "metrics-daemon-secret" not found
Sep 11 13:53:36.867696 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.867677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/513fe987-6474-4973-b14e-14020d75bdd8-original-pull-secret\") pod \"global-pull-secret-syncer-fp4jl\" (UID: \"513fe987-6474-4973-b14e-14020d75bdd8\") " pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:36.955217 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.955174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:36.958131 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.958108    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:53:36.967986 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.967960    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:53:36.978364 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:36.978337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vkk\" (UniqueName: \"kubernetes.io/projected/abcc8642-9a3b-4a54-aa55-ac49439b03f0-kube-api-access-v2vkk\") pod \"network-check-target-vx7x6\" (UID: \"abcc8642-9a3b-4a54-aa55-ac49439b03f0\") " pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:37.024081 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.024047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp4jl"
Sep 11 13:53:37.040145 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.040121    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-spk9p\""
Sep 11 13:53:37.048215 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.048195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:37.323260 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.323219    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fp4jl"]
Sep 11 13:53:37.326326 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:53:37.326300    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513fe987_6474_4973_b14e_14020d75bdd8.slice/crio-44e70415dec3bed033c44d861177c014a43eaa147f9a6680acbb5e7487105fda WatchSource:0}: Error finding container 44e70415dec3bed033c44d861177c014a43eaa147f9a6680acbb5e7487105fda: Status 404 returned error can't find the container with id 44e70415dec3bed033c44d861177c014a43eaa147f9a6680acbb5e7487105fda
Sep 11 13:53:37.356797 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.356738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vx7x6"]
Sep 11 13:53:37.357985 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.357848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:53:37.358043 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:37.358014    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:53:37.358043 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:37.358035    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:53:37.358111 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:37.358094    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.35807616 +0000 UTC m=+97.725665601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:53:37.360272 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:53:37.360248    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcc8642_9a3b_4a54_aa55_ac49439b03f0.slice/crio-eba75eb5cd81bee83dbc4a5c661228d770a21be23b9580a13ce13200fd01564b WatchSource:0}: Error finding container eba75eb5cd81bee83dbc4a5c661228d770a21be23b9580a13ce13200fd01564b: Status 404 returned error can't find the container with id eba75eb5cd81bee83dbc4a5c661228d770a21be23b9580a13ce13200fd01564b
Sep 11 13:53:37.459115 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.459034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:53:37.459261 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:37.459193    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:53:37.459307 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:53:37.459268    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.459244947 +0000 UTC m=+97.826834368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:53:37.481515 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.481485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vx7x6" event={"ID":"abcc8642-9a3b-4a54-aa55-ac49439b03f0","Type":"ContainerStarted","Data":"eba75eb5cd81bee83dbc4a5c661228d770a21be23b9580a13ce13200fd01564b"}
Sep 11 13:53:37.482675 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:37.482649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fp4jl" event={"ID":"513fe987-6474-4973-b14e-14020d75bdd8","Type":"ContainerStarted","Data":"44e70415dec3bed033c44d861177c014a43eaa147f9a6680acbb5e7487105fda"}
Sep 11 13:53:42.500700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:42.500606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vx7x6" event={"ID":"abcc8642-9a3b-4a54-aa55-ac49439b03f0","Type":"ContainerStarted","Data":"d575f2e6d5aa301382aab45d8e0a5f9e676bc0e78f20d97763180aa29b10a3fb"}
Sep 11 13:53:42.501194 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:42.500726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:53:42.502236 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:42.502216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fp4jl" event={"ID":"513fe987-6474-4973-b14e-14020d75bdd8","Type":"ContainerStarted","Data":"0725ce8eb1c698ae5b47ef42bfc41b2ca287f1ecb46bbf562add11b6cf76c2a5"}
Sep 11 13:53:42.517306 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:42.517264    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vx7x6" podStartSLOduration=65.741845498 podStartE2EDuration="1m10.517250351s" totalImagesPullingTime="4.775404853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:42.516667374 +0000 UTC m=+70.884256839" watchObservedRunningTime="2026-09-11 13:53:42.517250351 +0000 UTC m=+70.884839825"
Sep 11 13:53:42.532184 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:53:42.532146    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fp4jl" podStartSLOduration=65.71620061 podStartE2EDuration="1m10.532136679s" totalImagesPullingTime="4.815936069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:42.53189293 +0000 UTC m=+70.899482376" watchObservedRunningTime="2026-09-11 13:53:42.532136679 +0000 UTC m=+70.899726125"
Sep 11 13:54:08.608396 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.608359    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kf8c2"]
Sep 11 13:54:08.612591 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.612570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.615894 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.615874    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 13:54:08.617084 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.617064    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 13:54:08.617164 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.617099    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 13:54:08.617231 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.617100    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 13:54:08.617231 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.617101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-599v7\""
Sep 11 13:54:08.634827 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.634803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kf8c2"]
Sep 11 13:54:08.698398 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.698364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.698549 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.698401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj44g\" (UniqueName: \"kubernetes.io/projected/6725612c-a151-4d67-9b06-5bde83044106-kube-api-access-cj44g\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.698549 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.698437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6725612c-a151-4d67-9b06-5bde83044106-crio-socket\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.698549 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.698480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6725612c-a151-4d67-9b06-5bde83044106-data-volume\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.698549 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.698510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799324 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.799289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799498 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.799332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj44g\" (UniqueName: \"kubernetes.io/projected/6725612c-a151-4d67-9b06-5bde83044106-kube-api-access-cj44g\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799498 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.799353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6725612c-a151-4d67-9b06-5bde83044106-crio-socket\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799498 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.799379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6725612c-a151-4d67-9b06-5bde83044106-data-volume\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799498 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.799411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799498 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.799457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6725612c-a151-4d67-9b06-5bde83044106-crio-socket\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.799498 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:08.799460    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:08.799716 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:08.799536    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:54:08.799937 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:08.799889    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.299520923 +0000 UTC m=+97.667110358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:08.800093 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:08.800038    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.300015637 +0000 UTC m=+97.667605117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : configmap "kube-rbac-proxy" not found
Sep 11 13:54:08.800093 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.800043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6725612c-a151-4d67-9b06-5bde83044106-data-volume\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:08.810387 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:08.810362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj44g\" (UniqueName: \"kubernetes.io/projected/6725612c-a151-4d67-9b06-5bde83044106-kube-api-access-cj44g\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:09.303262 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:09.303209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:09.303451 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:09.303279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:09.303451 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.303360    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:09.303451 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.303408    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:54:09.303451 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.303438    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:10.303421187 +0000 UTC m=+98.671010609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:09.303585 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.303469    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:10.303456317 +0000 UTC m=+98.671045743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : configmap "kube-rbac-proxy" not found
Sep 11 13:54:09.404159 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:09.404113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") pod \"image-registry-598d5f97dd-wsrs7\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") " pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:54:09.404328 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.404278    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:54:09.404328 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.404304    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-598d5f97dd-wsrs7: secret "image-registry-tls" not found
Sep 11 13:54:09.404397 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.404363    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls podName:b618c1b1-c6b9-4751-9ad9-63d160f97e94 nodeName:}" failed. No retries permitted until 2026-09-11 13:55:13.40434829 +0000 UTC m=+161.771937714 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls") pod "image-registry-598d5f97dd-wsrs7" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94") : secret "image-registry-tls" not found
Sep 11 13:54:09.505537 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:09.505489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:54:09.505681 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.505644    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:54:09.505732 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:09.505721    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls podName:b9d9a6af-284c-4150-926d-f0cd36705cba nodeName:}" failed. No retries permitted until 2026-09-11 13:55:13.50570477 +0000 UTC m=+161.873294192 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls") pod "dns-default-dsn9c" (UID: "b9d9a6af-284c-4150-926d-f0cd36705cba") : secret "dns-default-metrics-tls" not found
Sep 11 13:54:10.310747 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:10.310709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:10.311187 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:10.310791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:10.311187 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:10.310835    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:54:10.311187 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:10.310881    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:10.311187 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:10.310927    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:12.310889123 +0000 UTC m=+100.678478544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : configmap "kube-rbac-proxy" not found
Sep 11 13:54:10.311187 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:10.310946    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:12.3109395 +0000 UTC m=+100.678528921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:11.136347 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:11.136321    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7h9r7_d45bb637-dafd-45e8-95dd-3731ba76f452/dns-node-resolver/0.log"
Sep 11 13:54:12.135794 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:12.135767    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g4phg_c5373e77-d971-4400-b35d-cd4bbe543ae5/node-ca/0.log"
Sep 11 13:54:12.328045 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:12.328005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:12.328209 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:12.328057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:12.328209 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:12.328143    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:54:12.328209 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:12.328143    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:12.328209 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:12.328194    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:16.328182196 +0000 UTC m=+104.695771618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : configmap "kube-rbac-proxy" not found
Sep 11 13:54:12.328342 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:12.328217    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:16.328210521 +0000 UTC m=+104.695799941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:12.935567 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:12.935537    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fz5x6_36bcb94d-4a3b-476a-b64a-8531b0e398f5/serve-healthcheck-canary/0.log"
Sep 11 13:54:13.507814 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:13.507785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vx7x6"
Sep 11 13:54:16.360048 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:16.360000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:16.360048 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:16.360063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:16.360496 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:16.360143    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:54:16.360496 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:16.360178    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:16.360496 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:16.360196    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:24.360181781 +0000 UTC m=+112.727771205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : configmap "kube-rbac-proxy" not found
Sep 11 13:54:16.360496 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:16.360237    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls podName:6725612c-a151-4d67-9b06-5bde83044106 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:24.360223294 +0000 UTC m=+112.727812715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kf8c2" (UID: "6725612c-a151-4d67-9b06-5bde83044106") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:54:24.422475 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:24.422438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:24.422851 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:24.422484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:24.422944 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:24.422904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6725612c-a151-4d67-9b06-5bde83044106-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:24.424663 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:24.424639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6725612c-a151-4d67-9b06-5bde83044106-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf8c2\" (UID: \"6725612c-a151-4d67-9b06-5bde83044106\") " pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:24.521123 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:24.521093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kf8c2"
Sep 11 13:54:24.793931 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:24.793836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kf8c2"]
Sep 11 13:54:24.797402 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:54:24.797372    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6725612c_a151_4d67_9b06_5bde83044106.slice/crio-ae345e4172b10f986036b866fb3af96c6e7087776bb2f59b571a6b57d59f7b27 WatchSource:0}: Error finding container ae345e4172b10f986036b866fb3af96c6e7087776bb2f59b571a6b57d59f7b27: Status 404 returned error can't find the container with id ae345e4172b10f986036b866fb3af96c6e7087776bb2f59b571a6b57d59f7b27
Sep 11 13:54:25.629616 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:25.629585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf8c2" event={"ID":"6725612c-a151-4d67-9b06-5bde83044106","Type":"ContainerStarted","Data":"cfc4afa9d009b7d0c1baa6e53f5f9476d6f5e9bdf4234aca068b6905eb0a240b"}
Sep 11 13:54:25.629995 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:25.629638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf8c2" event={"ID":"6725612c-a151-4d67-9b06-5bde83044106","Type":"ContainerStarted","Data":"f54e3d8e63d43fd1ae546b7f224ba409780993a47b5ad9b2125b80cb6b6a6b99"}
Sep 11 13:54:25.629995 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:25.629657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf8c2" event={"ID":"6725612c-a151-4d67-9b06-5bde83044106","Type":"ContainerStarted","Data":"ae345e4172b10f986036b866fb3af96c6e7087776bb2f59b571a6b57d59f7b27"}
Sep 11 13:54:27.637868 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:27.637833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf8c2" event={"ID":"6725612c-a151-4d67-9b06-5bde83044106","Type":"ContainerStarted","Data":"13c203147e4cbe1e0460aca79e17ca03ba0d6425207739e0c2f275b1065ea305"}
Sep 11 13:54:27.661482 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:27.661429    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kf8c2" podStartSLOduration=17.469663968 podStartE2EDuration="19.661414784s" totalImagesPullingTime="2.191750816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:54:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:27.660844863 +0000 UTC m=+116.028434308" watchObservedRunningTime="2026-09-11 13:54:27.661414784 +0000 UTC m=+116.029004228"
Sep 11 13:54:35.878466 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:35.878418    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" podUID="203ef1e2-d1a7-471d-938b-55440446c76e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 13:54:40.939434 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:40.939385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:54:40.941693 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:40.941674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70628814-f5ff-4599-a973-035afd41d226-metrics-certs\") pod \"network-metrics-daemon-rfb6q\" (UID: \"70628814-f5ff-4599-a973-035afd41d226\") " pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:54:41.234767 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:41.234688    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k94vs\""
Sep 11 13:54:41.241976 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:41.241955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfb6q"
Sep 11 13:54:41.512795 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:41.512716    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfb6q"]
Sep 11 13:54:41.515769 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:54:41.515735    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70628814_f5ff_4599_a973_035afd41d226.slice/crio-849013794f004a9d900c2c09d2358e2367a904e6b4bc37402417b249065e7af3 WatchSource:0}: Error finding container 849013794f004a9d900c2c09d2358e2367a904e6b4bc37402417b249065e7af3: Status 404 returned error can't find the container with id 849013794f004a9d900c2c09d2358e2367a904e6b4bc37402417b249065e7af3
Sep 11 13:54:41.677715 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:41.677677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfb6q" event={"ID":"70628814-f5ff-4599-a973-035afd41d226","Type":"ContainerStarted","Data":"849013794f004a9d900c2c09d2358e2367a904e6b4bc37402417b249065e7af3"}
Sep 11 13:54:42.682042 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:42.682014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfb6q" event={"ID":"70628814-f5ff-4599-a973-035afd41d226","Type":"ContainerStarted","Data":"a4edc913579d3c04f45965627ea6e4fa531b57a2df2212e48295ee04b06ca357"}
Sep 11 13:54:43.686611 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:43.686574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfb6q" event={"ID":"70628814-f5ff-4599-a973-035afd41d226","Type":"ContainerStarted","Data":"d5051665ade47ee223b4e5626c7ba87e00e8ae271c9f41ba28a6ebc92ca4d426"}
Sep 11 13:54:43.710035 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:43.709987    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rfb6q" podStartSLOduration=130.627816771 podStartE2EDuration="2m11.709972726s" totalImagesPullingTime="1.082155955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:43.709236559 +0000 UTC m=+132.076826002" watchObservedRunningTime="2026-09-11 13:54:43.709972726 +0000 UTC m=+132.077562196"
Sep 11 13:54:44.607028 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.606994    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k5cwr"]
Sep 11 13:54:44.610867 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.610845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.615595 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.615574    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 13:54:44.615867 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.615845    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 13:54:44.616049 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.616028    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 13:54:44.616825 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.616806    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 13:54:44.616935 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.616830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6tn2c\""
Sep 11 13:54:44.617008 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.616905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 13:54:44.617346 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.617130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 13:54:44.666961 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.666928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667206 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.666966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-sys\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667206 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.666992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d063270-9043-4db1-b7fc-eadb4dc7be0d-metrics-client-ca\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667206 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.667014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667206 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.667051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-textfile\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667206 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.667075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-wtmp\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667206 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.667135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-root\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667449 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.667215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.667449 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.667243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsj9\" (UniqueName: \"kubernetes.io/projected/1d063270-9043-4db1-b7fc-eadb4dc7be0d-kube-api-access-phsj9\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.767557 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-textfile\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.767557 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-wtmp\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-root\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsj9\" (UniqueName: \"kubernetes.io/projected/1d063270-9043-4db1-b7fc-eadb4dc7be0d-kube-api-access-phsj9\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-root\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-sys\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:44.767953    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-textfile\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.767972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d063270-9043-4db1-b7fc-eadb4dc7be0d-metrics-client-ca\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:44.768023    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls podName:1d063270-9043-4db1-b7fc-eadb4dc7be0d nodeName:}" failed. No retries permitted until 2026-09-11 13:54:45.268004161 +0000 UTC m=+133.635593586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls") pod "node-exporter-k5cwr" (UID: "1d063270-9043-4db1-b7fc-eadb4dc7be0d") : secret "node-exporter-tls" not found
Sep 11 13:54:44.768037 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.768025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-sys\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768478 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.768074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768478 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.768101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-wtmp\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768478 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.768429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.768478 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.768453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d063270-9043-4db1-b7fc-eadb4dc7be0d-metrics-client-ca\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.770250 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.770231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:44.783819 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:44.783791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsj9\" (UniqueName: \"kubernetes.io/projected/1d063270-9043-4db1-b7fc-eadb4dc7be0d-kube-api-access-phsj9\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:45.273091 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:45.273048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:45.273284 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:45.273174    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 13:54:45.273284 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:45.273257    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls podName:1d063270-9043-4db1-b7fc-eadb4dc7be0d nodeName:}" failed. No retries permitted until 2026-09-11 13:54:46.273241934 +0000 UTC m=+134.640831368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls") pod "node-exporter-k5cwr" (UID: "1d063270-9043-4db1-b7fc-eadb4dc7be0d") : secret "node-exporter-tls" not found
Sep 11 13:54:45.878291 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:45.878251    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" podUID="203ef1e2-d1a7-471d-938b-55440446c76e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 13:54:46.280832 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:46.280745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:46.283060 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:46.283031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d063270-9043-4db1-b7fc-eadb4dc7be0d-node-exporter-tls\") pod \"node-exporter-k5cwr\" (UID: \"1d063270-9043-4db1-b7fc-eadb4dc7be0d\") " pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:46.420017 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:46.419978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k5cwr"
Sep 11 13:54:46.430820 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:54:46.430794    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d063270_9043_4db1_b7fc_eadb4dc7be0d.slice/crio-2b65ffb41c12a6cf9fa17823a82040c12265dbaefd1ca2a3950a54b4944e6e90 WatchSource:0}: Error finding container 2b65ffb41c12a6cf9fa17823a82040c12265dbaefd1ca2a3950a54b4944e6e90: Status 404 returned error can't find the container with id 2b65ffb41c12a6cf9fa17823a82040c12265dbaefd1ca2a3950a54b4944e6e90
Sep 11 13:54:46.696922 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:46.696871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5cwr" event={"ID":"1d063270-9043-4db1-b7fc-eadb4dc7be0d","Type":"ContainerStarted","Data":"2b65ffb41c12a6cf9fa17823a82040c12265dbaefd1ca2a3950a54b4944e6e90"}
Sep 11 13:54:47.701866 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:47.701832    2561 generic.go:356] "Generic (PLEG): container finished" podID="1d063270-9043-4db1-b7fc-eadb4dc7be0d" containerID="5aa6db57920a6a6a4ee6ae0a129fb746db5f1f79b6e6cfae495b2775f594f1ff" exitCode=0
Sep 11 13:54:47.702300 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:47.701948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5cwr" event={"ID":"1d063270-9043-4db1-b7fc-eadb4dc7be0d","Type":"ContainerDied","Data":"5aa6db57920a6a6a4ee6ae0a129fb746db5f1f79b6e6cfae495b2775f594f1ff"}
Sep 11 13:54:48.707630 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:48.707596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5cwr" event={"ID":"1d063270-9043-4db1-b7fc-eadb4dc7be0d","Type":"ContainerStarted","Data":"f8a2bfeb95df6c6c74b31a7d6d4a183d51a78ac86d4f63fba3585a48c15261e7"}
Sep 11 13:54:48.708029 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:48.707650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5cwr" event={"ID":"1d063270-9043-4db1-b7fc-eadb4dc7be0d","Type":"ContainerStarted","Data":"01866e1e4a607abc07fcb514fd8126b1105aef7261ecb035b71bbb04596ec7f4"}
Sep 11 13:54:48.735289 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:48.735243    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k5cwr" podStartSLOduration=3.638640296 podStartE2EDuration="4.735227192s" totalImagesPullingTime="1.035705581s" totalInitContainerRuntime="60.881315ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:48.734050114 +0000 UTC m=+137.101639557" watchObservedRunningTime="2026-09-11 13:54:48.735227192 +0000 UTC m=+137.102816634"
Sep 11 13:54:55.194011 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.193980    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-598d5f97dd-wsrs7"]
Sep 11 13:54:55.194390 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:54:55.194182    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7" podUID="b618c1b1-c6b9-4751-9ad9-63d160f97e94"
Sep 11 13:54:55.727891 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.727861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:54:55.734074 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.734052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:54:55.851048 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851014    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-certificates\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851219 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-installation-pull-secrets\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851219 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b618c1b1-c6b9-4751-9ad9-63d160f97e94-ca-trust-extracted\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851219 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851133    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ft28\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-kube-api-access-6ft28\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851219 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851175    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-bound-sa-token\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851219 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-trusted-ca\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851473 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851232    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-image-registry-private-configuration\") pod \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\" (UID: \"b618c1b1-c6b9-4751-9ad9-63d160f97e94\") "
Sep 11 13:54:55.851473 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:54:55.851473 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851440    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-certificates\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:55.851642 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b618c1b1-c6b9-4751-9ad9-63d160f97e94-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:54:55.851756 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.851727    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:54:55.853524 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.853503    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:54:55.853640 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.853593    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:54:55.853696 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.853659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-kube-api-access-6ft28" (OuterVolumeSpecName: "kube-api-access-6ft28") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "kube-api-access-6ft28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:54:55.853806 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.853786    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b618c1b1-c6b9-4751-9ad9-63d160f97e94" (UID: "b618c1b1-c6b9-4751-9ad9-63d160f97e94"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:54:55.878809 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.878769    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" podUID="203ef1e2-d1a7-471d-938b-55440446c76e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 13:54:55.878967 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.878840    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf"
Sep 11 13:54:55.879323 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.879294    2561 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"841ee4dc1100b3bdeff7c78ecb55cc825bbbb050c8118a71f493c7508159f02e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 13:54:55.879367 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.879344    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" podUID="203ef1e2-d1a7-471d-938b-55440446c76e" containerName="service-proxy" containerID="cri-o://841ee4dc1100b3bdeff7c78ecb55cc825bbbb050c8118a71f493c7508159f02e" gracePeriod=30
Sep 11 13:54:55.952824 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.952785    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b618c1b1-c6b9-4751-9ad9-63d160f97e94-trusted-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:55.952824 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.952819    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-image-registry-private-configuration\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:55.953080 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.952834    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b618c1b1-c6b9-4751-9ad9-63d160f97e94-installation-pull-secrets\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:55.953080 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.952846    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b618c1b1-c6b9-4751-9ad9-63d160f97e94-ca-trust-extracted\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:55.953080 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.952855    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ft28\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-kube-api-access-6ft28\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:55.953080 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:55.952865    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-bound-sa-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:56.733147 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.733114    2561 generic.go:356] "Generic (PLEG): container finished" podID="203ef1e2-d1a7-471d-938b-55440446c76e" containerID="841ee4dc1100b3bdeff7c78ecb55cc825bbbb050c8118a71f493c7508159f02e" exitCode=2
Sep 11 13:54:56.733544 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.733191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" event={"ID":"203ef1e2-d1a7-471d-938b-55440446c76e","Type":"ContainerDied","Data":"841ee4dc1100b3bdeff7c78ecb55cc825bbbb050c8118a71f493c7508159f02e"}
Sep 11 13:54:56.733544 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.733227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-598d5f97dd-wsrs7"
Sep 11 13:54:56.733544 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.733247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cd95f7c6-vjkwf" event={"ID":"203ef1e2-d1a7-471d-938b-55440446c76e","Type":"ContainerStarted","Data":"d431a49a1d2fe2f178b8fe16041d54f4336f79b3515f4822c3b3174cffb369fd"}
Sep 11 13:54:56.796971 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.796940    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-598d5f97dd-wsrs7"]
Sep 11 13:54:56.801156 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.801129    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-598d5f97dd-wsrs7"]
Sep 11 13:54:56.859204 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:56.859178    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b618c1b1-c6b9-4751-9ad9-63d160f97e94-registry-tls\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:54:58.115418 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:54:58.115386    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b618c1b1-c6b9-4751-9ad9-63d160f97e94" path="/var/lib/kubelet/pods/b618c1b1-c6b9-4751-9ad9-63d160f97e94/volumes"
Sep 11 13:55:08.604880 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:55:08.604834    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dsn9c" podUID="b9d9a6af-284c-4150-926d-f0cd36705cba"
Sep 11 13:55:08.770818 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:08.770786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:55:13.583384 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:13.583343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:55:13.585700 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:13.585678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d9a6af-284c-4150-926d-f0cd36705cba-metrics-tls\") pod \"dns-default-dsn9c\" (UID: \"b9d9a6af-284c-4150-926d-f0cd36705cba\") " pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:55:13.874409 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:13.874329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nllbd\""
Sep 11 13:55:13.882189 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:13.882165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:55:14.156348 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:14.156280    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dsn9c"]
Sep 11 13:55:14.158947 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:55:14.158919    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d9a6af_284c_4150_926d_f0cd36705cba.slice/crio-dab9d151df283f60c99742b8da495bdc8422b5398feb0aa2fd591345d1a22852 WatchSource:0}: Error finding container dab9d151df283f60c99742b8da495bdc8422b5398feb0aa2fd591345d1a22852: Status 404 returned error can't find the container with id dab9d151df283f60c99742b8da495bdc8422b5398feb0aa2fd591345d1a22852
Sep 11 13:55:14.790684 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:14.790647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsn9c" event={"ID":"b9d9a6af-284c-4150-926d-f0cd36705cba","Type":"ContainerStarted","Data":"dab9d151df283f60c99742b8da495bdc8422b5398feb0aa2fd591345d1a22852"}
Sep 11 13:55:16.798728 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:16.798695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsn9c" event={"ID":"b9d9a6af-284c-4150-926d-f0cd36705cba","Type":"ContainerStarted","Data":"866070fc1a9a14161ec4aee35495a910a2c4177efac0c4d6645f3552214ab3e6"}
Sep 11 13:55:16.799170 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:16.798745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dsn9c" event={"ID":"b9d9a6af-284c-4150-926d-f0cd36705cba","Type":"ContainerStarted","Data":"a502e7bc3c75081d1eb0e5fd3b26489c70e27061f29bda0e5b4441a4e7d5ac61"}
Sep 11 13:55:16.799170 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:16.798831    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:55:16.818603 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:16.818551    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dsn9c" podStartSLOduration=129.986139946 podStartE2EDuration="2m11.818535965s" totalImagesPullingTime="1.832396019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:16.818299274 +0000 UTC m=+165.185888722" watchObservedRunningTime="2026-09-11 13:55:16.818535965 +0000 UTC m=+165.186125434"
Sep 11 13:55:26.804475 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:55:26.804395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dsn9c"
Sep 11 13:57:32.058809 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:32.058777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 13:57:32.063287 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:32.063257    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 13:57:32.066129 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:32.066110    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 13:57:53.226971 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.226936    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"]
Sep 11 13:57:53.230196 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.230182    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.232842 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.232822    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 13:57:53.232937 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.232847    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 13:57:53.233861 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.233842    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kpc2b\""
Sep 11 13:57:53.246310 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.243928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"]
Sep 11 13:57:53.334852 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.334823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0a920d0-68c9-4b8d-b77d-3636224ac7a6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zzvtg\" (UID: \"a0a920d0-68c9-4b8d-b77d-3636224ac7a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.335014 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.334866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvkb\" (UniqueName: \"kubernetes.io/projected/a0a920d0-68c9-4b8d-b77d-3636224ac7a6-kube-api-access-cjvkb\") pod \"cert-manager-cainjector-559cdd8c87-zzvtg\" (UID: \"a0a920d0-68c9-4b8d-b77d-3636224ac7a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.435371 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.435338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvkb\" (UniqueName: \"kubernetes.io/projected/a0a920d0-68c9-4b8d-b77d-3636224ac7a6-kube-api-access-cjvkb\") pod \"cert-manager-cainjector-559cdd8c87-zzvtg\" (UID: \"a0a920d0-68c9-4b8d-b77d-3636224ac7a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.435499 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.435411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0a920d0-68c9-4b8d-b77d-3636224ac7a6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zzvtg\" (UID: \"a0a920d0-68c9-4b8d-b77d-3636224ac7a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.444053 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.444030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0a920d0-68c9-4b8d-b77d-3636224ac7a6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zzvtg\" (UID: \"a0a920d0-68c9-4b8d-b77d-3636224ac7a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.444544 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.444521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvkb\" (UniqueName: \"kubernetes.io/projected/a0a920d0-68c9-4b8d-b77d-3636224ac7a6-kube-api-access-cjvkb\") pod \"cert-manager-cainjector-559cdd8c87-zzvtg\" (UID: \"a0a920d0-68c9-4b8d-b77d-3636224ac7a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.538644 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.538558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"
Sep 11 13:57:53.805632 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.805556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg"]
Sep 11 13:57:53.808314 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:57:53.808286    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a920d0_68c9_4b8d_b77d_3636224ac7a6.slice/crio-464a9ca98c849de4d27f0253cc958c0b033190a41def351ffa9adea00d8c6e84 WatchSource:0}: Error finding container 464a9ca98c849de4d27f0253cc958c0b033190a41def351ffa9adea00d8c6e84: Status 404 returned error can't find the container with id 464a9ca98c849de4d27f0253cc958c0b033190a41def351ffa9adea00d8c6e84
Sep 11 13:57:53.810274 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:53.810256    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:57:54.288382 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:54.288341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg" event={"ID":"a0a920d0-68c9-4b8d-b77d-3636224ac7a6","Type":"ContainerStarted","Data":"464a9ca98c849de4d27f0253cc958c0b033190a41def351ffa9adea00d8c6e84"}
Sep 11 13:57:57.302403 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:57.302358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg" event={"ID":"a0a920d0-68c9-4b8d-b77d-3636224ac7a6","Type":"ContainerStarted","Data":"ff5e34148f30e05ff2b13eafba9ac74d6300c090e3762ad401cd76a6ec960308"}
Sep 11 13:57:57.318110 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:57:57.318054    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzvtg" podStartSLOduration=1.463687064 podStartE2EDuration="4.318039216s" totalImagesPullingTime="2.854352152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:57:57.317900204 +0000 UTC m=+325.685489649" watchObservedRunningTime="2026-09-11 13:57:57.318039216 +0000 UTC m=+325.685628660"
Sep 11 13:58:23.658628 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.658592    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"]
Sep 11 13:58:23.661696 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.661680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.666946 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.666902    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 13:58:23.666946 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.666939    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 13:58:23.667267 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.667248    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rkxmr\""
Sep 11 13:58:23.667315 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.667267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 13:58:23.667946 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.667929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 13:58:23.697581 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.697543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"]
Sep 11 13:58:23.755964 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.755927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92d26e17-3357-48e0-810e-357662a5615e-manifests\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.756127 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.755983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm29j\" (UniqueName: \"kubernetes.io/projected/92d26e17-3357-48e0-810e-357662a5615e-kube-api-access-vm29j\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.756127 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.756003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92d26e17-3357-48e0-810e-357662a5615e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.756127 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.756023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92d26e17-3357-48e0-810e-357662a5615e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.756127 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.756039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92d26e17-3357-48e0-810e-357662a5615e-tmp\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.856967 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.856903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm29j\" (UniqueName: \"kubernetes.io/projected/92d26e17-3357-48e0-810e-357662a5615e-kube-api-access-vm29j\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.856967 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.856977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92d26e17-3357-48e0-810e-357662a5615e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.857221 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.857009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92d26e17-3357-48e0-810e-357662a5615e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.857221 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.857034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92d26e17-3357-48e0-810e-357662a5615e-tmp\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.857221 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.857105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92d26e17-3357-48e0-810e-357662a5615e-manifests\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.857519 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.857488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92d26e17-3357-48e0-810e-357662a5615e-manifests\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.859459 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.859428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92d26e17-3357-48e0-810e-357662a5615e-tmp\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.859846 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.859828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92d26e17-3357-48e0-810e-357662a5615e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.859942 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.859877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92d26e17-3357-48e0-810e-357662a5615e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.867173 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.867150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm29j\" (UniqueName: \"kubernetes.io/projected/92d26e17-3357-48e0-810e-357662a5615e-kube-api-access-vm29j\") pod \"opendatahub-operator-controller-manager-7ff676d985-c4gdv\" (UID: \"92d26e17-3357-48e0-810e-357662a5615e\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:23.971845 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:23.971755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:24.247613 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:24.247515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"]
Sep 11 13:58:24.250619 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:58:24.250595    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d26e17_3357_48e0_810e_357662a5615e.slice/crio-9b92ca680023b8fba84db28e94b5d1770dafca4d88a49a9f6113379bf1030fef WatchSource:0}: Error finding container 9b92ca680023b8fba84db28e94b5d1770dafca4d88a49a9f6113379bf1030fef: Status 404 returned error can't find the container with id 9b92ca680023b8fba84db28e94b5d1770dafca4d88a49a9f6113379bf1030fef
Sep 11 13:58:24.390650 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:24.390615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv" event={"ID":"92d26e17-3357-48e0-810e-357662a5615e","Type":"ContainerStarted","Data":"9b92ca680023b8fba84db28e94b5d1770dafca4d88a49a9f6113379bf1030fef"}
Sep 11 13:58:27.404585 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:27.404502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv" event={"ID":"92d26e17-3357-48e0-810e-357662a5615e","Type":"ContainerStarted","Data":"c1d9a20f572a42b2f6cae8225c5ebf3127bcbccea7ff9e155e409a75dfa20477"}
Sep 11 13:58:28.409236 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:28.409199    2561 generic.go:356] "Generic (PLEG): container finished" podID="92d26e17-3357-48e0-810e-357662a5615e" containerID="c1d9a20f572a42b2f6cae8225c5ebf3127bcbccea7ff9e155e409a75dfa20477" exitCode=0
Sep 11 13:58:28.409632 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:28.409290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv" event={"ID":"92d26e17-3357-48e0-810e-357662a5615e","Type":"ContainerDied","Data":"c1d9a20f572a42b2f6cae8225c5ebf3127bcbccea7ff9e155e409a75dfa20477"}
Sep 11 13:58:29.414621 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:29.414586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv" event={"ID":"92d26e17-3357-48e0-810e-357662a5615e","Type":"ContainerStarted","Data":"d276ef118b8fed3391a2792fc4b44652787d0d8a8536fa7556ee64eb00b4c287"}
Sep 11 13:58:29.415049 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:29.414720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:29.438916 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:29.438872    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv" podStartSLOduration=2.129843388 podStartE2EDuration="6.438856773s" totalImagesPullingTime="3.242361352s" totalInitContainerRuntime="1.066652033s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:29.438766199 +0000 UTC m=+357.806355643" watchObservedRunningTime="2026-09-11 13:58:29.438856773 +0000 UTC m=+357.806446215"
Sep 11 13:58:40.421024 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:40.420994    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-c4gdv"
Sep 11 13:58:43.681721 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.681686    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"]
Sep 11 13:58:43.693221 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.693200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.697364 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.697344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 13:58:43.697643 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.697622    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 13:58:43.698107 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.698093    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 13:58:43.698630 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.698616    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kgtlx\""
Sep 11 13:58:43.707271 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.707249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"]
Sep 11 13:58:43.738947 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.738924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 13:58:43.806581 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.806556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.806690 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.806598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tmp\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.806690 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.806623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtw2p\" (UniqueName: \"kubernetes.io/projected/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-kube-api-access-jtw2p\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.907292 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.907265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.907403 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.907307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tmp\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.907403 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.907344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtw2p\" (UniqueName: \"kubernetes.io/projected/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-kube-api-access-jtw2p\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.907473 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:58:43.907395    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 13:58:43.907506 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:58:43.907472    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tls-certs podName:e88d754d-c6e7-4bc6-8f7b-9bde58c5e995 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:44.407455084 +0000 UTC m=+372.775044516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tls-certs") pod "kube-auth-proxy-746c8b5fff-jpqm6" (UID: "e88d754d-c6e7-4bc6-8f7b-9bde58c5e995") : secret "kube-auth-proxy-tls" not found
Sep 11 13:58:43.909544 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.909526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tmp\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:43.935606 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:43.935546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtw2p\" (UniqueName: \"kubernetes.io/projected/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-kube-api-access-jtw2p\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:44.411181 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:44.411146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:44.413875 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:44.413841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88d754d-c6e7-4bc6-8f7b-9bde58c5e995-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-jpqm6\" (UID: \"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:44.601799 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:44.601761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"
Sep 11 13:58:44.873593 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:44.873567    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6"]
Sep 11 13:58:44.875427 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:58:44.875402    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88d754d_c6e7_4bc6_8f7b_9bde58c5e995.slice/crio-a98ed67c99d86fc2351f216451f5c1285a384aaddeab615caca3244d05073cb8 WatchSource:0}: Error finding container a98ed67c99d86fc2351f216451f5c1285a384aaddeab615caca3244d05073cb8: Status 404 returned error can't find the container with id a98ed67c99d86fc2351f216451f5c1285a384aaddeab615caca3244d05073cb8
Sep 11 13:58:45.105522 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.105485    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"]
Sep 11 13:58:45.109843 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.109820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.114005 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.113984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 13:58:45.114107 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.113988    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 13:58:45.114107 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.114091    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 13:58:45.115273 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.115199    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8pprv\""
Sep 11 13:58:45.115273 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.115225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 13:58:45.129863 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.129789    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"]
Sep 11 13:58:45.216414 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.216586 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677946e3-f877-416b-89fd-afdee6d1f8b0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.216586 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpp4\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-kube-api-access-xqpp4\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.216586 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.216692 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.216692 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.216692 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.216639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.317691 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.317691 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.318007 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.318007 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.318007 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.318007 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677946e3-f877-416b-89fd-afdee6d1f8b0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.318007 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.317803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpp4\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-kube-api-access-xqpp4\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.318410 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.318356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.320256 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.320228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.320351 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.320277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677946e3-f877-416b-89fd-afdee6d1f8b0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.320397 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.320385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.320432 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.320409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.327617 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.327592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.327888 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.327870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpp4\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-kube-api-access-xqpp4\") pod \"istiod-openshift-gateway-7b7995d5cd-27gxt\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.418752 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.418639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:45.473635 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.473601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6" event={"ID":"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995","Type":"ContainerStarted","Data":"a98ed67c99d86fc2351f216451f5c1285a384aaddeab615caca3244d05073cb8"}
Sep 11 13:58:45.700195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:45.700111    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"]
Sep 11 13:58:45.703612 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:58:45.703558    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677946e3_f877_416b_89fd_afdee6d1f8b0.slice/crio-d4af640bb0701b36467ce353739771f1091039a4e3525fb674a0cec473bfa40b WatchSource:0}: Error finding container d4af640bb0701b36467ce353739771f1091039a4e3525fb674a0cec473bfa40b: Status 404 returned error can't find the container with id d4af640bb0701b36467ce353739771f1091039a4e3525fb674a0cec473bfa40b
Sep 11 13:58:46.479352 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:46.479310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" event={"ID":"677946e3-f877-416b-89fd-afdee6d1f8b0","Type":"ContainerStarted","Data":"d4af640bb0701b36467ce353739771f1091039a4e3525fb674a0cec473bfa40b"}
Sep 11 13:58:48.488039 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:48.487997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6" event={"ID":"e88d754d-c6e7-4bc6-8f7b-9bde58c5e995","Type":"ContainerStarted","Data":"db3807ef6919316143418fd9d3b5e217816fcebb620821b72e4f3bdc725e72a0"}
Sep 11 13:58:48.510409 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:48.510365    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-jpqm6" podStartSLOduration=2.849460783 podStartE2EDuration="5.510349315s" totalImagesPullingTime="2.660888532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:48.509003065 +0000 UTC m=+376.876592509" watchObservedRunningTime="2026-09-11 13:58:48.510349315 +0000 UTC m=+376.877938758"
Sep 11 13:58:49.290373 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:49.290337    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 13:58:49.290454 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:49.290422    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 13:58:49.493748 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:49.493711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" event={"ID":"677946e3-f877-416b-89fd-afdee6d1f8b0","Type":"ContainerStarted","Data":"70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53"}
Sep 11 13:58:49.494225 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:49.493929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:58:49.518516 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:49.518475    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" podStartSLOduration=0.934872607 podStartE2EDuration="4.518457722s" totalImagesPullingTime="3.583585115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:49.517153506 +0000 UTC m=+377.884742949" watchObservedRunningTime="2026-09-11 13:58:49.518457722 +0000 UTC m=+377.886047167"
Sep 11 13:58:50.498494 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:50.498457    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-27gxt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 13:58:50.498867 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:50.498522    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" podUID="677946e3-f877-416b-89fd-afdee6d1f8b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:58:53.497869 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:58:53.497835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:59:01.246647 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.246615    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"]
Sep 11 13:59:01.254848 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.254829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.258527 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.258499    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 13:59:01.258797 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.258771    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 13:59:01.259541 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.259523    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 13:59:01.263310 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.263286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"]
Sep 11 13:59:01.272608 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.272592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:59:01.275641 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.275626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j6xsm\""
Sep 11 13:59:01.275741 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.275633    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 13:59:01.344925 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.344877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfc8\" (UniqueName: \"kubernetes.io/projected/3f0745ad-efa0-42c3-8a49-70d6640b20f1-kube-api-access-ttfc8\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.345115 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.344950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0745ad-efa0-42c3-8a49-70d6640b20f1-metrics-cert\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.345115 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.344980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0745ad-efa0-42c3-8a49-70d6640b20f1-cert\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.345115 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.344997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3f0745ad-efa0-42c3-8a49-70d6640b20f1-manager-config\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.446265 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.446222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfc8\" (UniqueName: \"kubernetes.io/projected/3f0745ad-efa0-42c3-8a49-70d6640b20f1-kube-api-access-ttfc8\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.446423 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.446278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0745ad-efa0-42c3-8a49-70d6640b20f1-metrics-cert\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.446423 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.446315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0745ad-efa0-42c3-8a49-70d6640b20f1-cert\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.446423 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.446337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3f0745ad-efa0-42c3-8a49-70d6640b20f1-manager-config\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.446992 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.446970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3f0745ad-efa0-42c3-8a49-70d6640b20f1-manager-config\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.448920 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.448893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0745ad-efa0-42c3-8a49-70d6640b20f1-cert\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.448984 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.448962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f0745ad-efa0-42c3-8a49-70d6640b20f1-metrics-cert\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.455149 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.455127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfc8\" (UniqueName: \"kubernetes.io/projected/3f0745ad-efa0-42c3-8a49-70d6640b20f1-kube-api-access-ttfc8\") pod \"lws-controller-manager-594d9cdff7-96j95\" (UID: \"3f0745ad-efa0-42c3-8a49-70d6640b20f1\") " pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:01.563734 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:01.563705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:02.231152 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:02.231119    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"]
Sep 11 13:59:02.233639 ip-10-0-128-105 kubenswrapper[2561]: W0911 13:59:02.233611    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0745ad_efa0_42c3_8a49_70d6640b20f1.slice/crio-8827857bc99a574a0fbe3e6fce8680e1b5acf541a533503a6a75fac337fcfa79 WatchSource:0}: Error finding container 8827857bc99a574a0fbe3e6fce8680e1b5acf541a533503a6a75fac337fcfa79: Status 404 returned error can't find the container with id 8827857bc99a574a0fbe3e6fce8680e1b5acf541a533503a6a75fac337fcfa79
Sep 11 13:59:02.539608 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:02.539528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95" event={"ID":"3f0745ad-efa0-42c3-8a49-70d6640b20f1","Type":"ContainerStarted","Data":"8827857bc99a574a0fbe3e6fce8680e1b5acf541a533503a6a75fac337fcfa79"}
Sep 11 13:59:05.687421 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.687388    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"]
Sep 11 13:59:05.687850 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.687646    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" podUID="677946e3-f877-416b-89fd-afdee6d1f8b0" containerName="discovery" containerID="cri-o://70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53" gracePeriod=30
Sep 11 13:59:05.931670 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.931648    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:59:05.979033 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.978979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677946e3-f877-416b-89fd-afdee6d1f8b0-local-certs\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979033 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-cacerts\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979053    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-dns-cert\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-ca-configmap\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979097    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-kubeconfig\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqpp4\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-kube-api-access-xqpp4\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979195 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979153    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-token\") pod \"677946e3-f877-416b-89fd-afdee6d1f8b0\" (UID: \"677946e3-f877-416b-89fd-afdee6d1f8b0\") "
Sep 11 13:59:05.979574 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.979546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:59:05.981825 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.981785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:05.981825 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.981805    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-cacerts" (OuterVolumeSpecName: "cacerts") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:05.981999 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.981855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:05.981999 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.981855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677946e3-f877-416b-89fd-afdee6d1f8b0-local-certs" (OuterVolumeSpecName: "local-certs") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:59:05.981999 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.981888    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:05.981999 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:05.981897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-kube-api-access-xqpp4" (OuterVolumeSpecName: "kube-api-access-xqpp4") pod "677946e3-f877-416b-89fd-afdee6d1f8b0" (UID: "677946e3-f877-416b-89fd-afdee6d1f8b0"). InnerVolumeSpecName "kube-api-access-xqpp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:06.080314 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080280    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqpp4\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-kube-api-access-xqpp4\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.080314 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080308    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.080314 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080321    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677946e3-f877-416b-89fd-afdee6d1f8b0-local-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.080531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080332    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-cacerts\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.080531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080343    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-dns-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.080531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080354    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-csr-ca-configmap\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.080531 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.080364    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677946e3-f877-416b-89fd-afdee6d1f8b0-istio-kubeconfig\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 13:59:06.554553 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.554520    2561 generic.go:356] "Generic (PLEG): container finished" podID="677946e3-f877-416b-89fd-afdee6d1f8b0" containerID="70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53" exitCode=0
Sep 11 13:59:06.554726 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.554570    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"
Sep 11 13:59:06.554726 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.554574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" event={"ID":"677946e3-f877-416b-89fd-afdee6d1f8b0","Type":"ContainerDied","Data":"70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53"}
Sep 11 13:59:06.554726 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.554699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt" event={"ID":"677946e3-f877-416b-89fd-afdee6d1f8b0","Type":"ContainerDied","Data":"d4af640bb0701b36467ce353739771f1091039a4e3525fb674a0cec473bfa40b"}
Sep 11 13:59:06.554835 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.554737    2561 scope.go:117] "RemoveContainer" containerID="70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53"
Sep 11 13:59:06.563342 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.563326    2561 scope.go:117] "RemoveContainer" containerID="70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53"
Sep 11 13:59:06.563614 ip-10-0-128-105 kubenswrapper[2561]: E0911 13:59:06.563596    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53\": container with ID starting with 70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53 not found: ID does not exist" containerID="70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53"
Sep 11 13:59:06.563669 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.563622    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53"} err="failed to get container status \"70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53\": rpc error: code = NotFound desc = could not find container \"70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53\": container with ID starting with 70ed9037df3ebcfa68a4ed328a7045eb4a8e11984223af653cf5f356a74a5a53 not found: ID does not exist"
Sep 11 13:59:06.594364 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.594341    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"]
Sep 11 13:59:06.631633 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:06.631609    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-27gxt"]
Sep 11 13:59:08.115320 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:08.115288    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677946e3-f877-416b-89fd-afdee6d1f8b0" path="/var/lib/kubelet/pods/677946e3-f877-416b-89fd-afdee6d1f8b0/volumes"
Sep 11 13:59:16.593401 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:16.593369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95" event={"ID":"3f0745ad-efa0-42c3-8a49-70d6640b20f1","Type":"ContainerStarted","Data":"4e85c158b0b09d3629aae77f673db5d27f1b147f065e40ffaab66c73a49c20d6"}
Sep 11 13:59:16.593812 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:16.593502    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 13:59:16.614597 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:16.614538    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95" podStartSLOduration=1.599557474 podStartE2EDuration="15.614519965s" totalImagesPullingTime="14.014962491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:16.614067927 +0000 UTC m=+404.981657393" watchObservedRunningTime="2026-09-11 13:59:16.614519965 +0000 UTC m=+404.982109423"
Sep 11 13:59:27.599588 ip-10-0-128-105 kubenswrapper[2561]: I0911 13:59:27.599557    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-594d9cdff7-96j95"
Sep 11 14:00:23.416881 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.416847    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"]
Sep 11 14:00:23.417326 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.417102    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677946e3-f877-416b-89fd-afdee6d1f8b0" containerName="discovery"
Sep 11 14:00:23.417326 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.417114    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="677946e3-f877-416b-89fd-afdee6d1f8b0" containerName="discovery"
Sep 11 14:00:23.417326 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.417182    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="677946e3-f877-416b-89fd-afdee6d1f8b0" containerName="discovery"
Sep 11 14:00:23.420004 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.419989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.422537 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.422513    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 14:00:23.422663 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.422540    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 14:00:23.422764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.422751    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zlqsx\""
Sep 11 14:00:23.423586 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.423561    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 14:00:23.423706 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.423574    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 14:00:23.430941 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.430904    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"]
Sep 11 14:00:23.462818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.462789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft882\" (UniqueName: \"kubernetes.io/projected/c48d1677-1c7f-412c-a532-f870ebb388a2-kube-api-access-ft882\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.462995 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.462840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c48d1677-1c7f-412c-a532-f870ebb388a2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.462995 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.462893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c48d1677-1c7f-412c-a532-f870ebb388a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.563665 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.563633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c48d1677-1c7f-412c-a532-f870ebb388a2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.563848 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.563676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c48d1677-1c7f-412c-a532-f870ebb388a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.563848 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.563709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft882\" (UniqueName: \"kubernetes.io/projected/c48d1677-1c7f-412c-a532-f870ebb388a2-kube-api-access-ft882\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.563950 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:00:23.563845    2561 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 14:00:23.563950 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:00:23.563927    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c48d1677-1c7f-412c-a532-f870ebb388a2-plugin-serving-cert podName:c48d1677-1c7f-412c-a532-f870ebb388a2 nodeName:}" failed. No retries permitted until 2026-09-11 14:00:24.063891336 +0000 UTC m=+472.431480771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c48d1677-1c7f-412c-a532-f870ebb388a2-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-j6mbw" (UID: "c48d1677-1c7f-412c-a532-f870ebb388a2") : secret "plugin-serving-cert" not found
Sep 11 14:00:23.564358 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.564339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c48d1677-1c7f-412c-a532-f870ebb388a2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:23.575048 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:23.575025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft882\" (UniqueName: \"kubernetes.io/projected/c48d1677-1c7f-412c-a532-f870ebb388a2-kube-api-access-ft882\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:24.067000 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:24.066961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c48d1677-1c7f-412c-a532-f870ebb388a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:24.069444 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:24.069410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c48d1677-1c7f-412c-a532-f870ebb388a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-j6mbw\" (UID: \"c48d1677-1c7f-412c-a532-f870ebb388a2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:24.329195 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:24.329094    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"
Sep 11 14:00:24.602900 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:24.602776    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw"]
Sep 11 14:00:24.605353 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:00:24.605326    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc48d1677_1c7f_412c_a532_f870ebb388a2.slice/crio-d743f651108141d13d099b6d11022182387ffd4bee54c29f01106ae21a181a52 WatchSource:0}: Error finding container d743f651108141d13d099b6d11022182387ffd4bee54c29f01106ae21a181a52: Status 404 returned error can't find the container with id d743f651108141d13d099b6d11022182387ffd4bee54c29f01106ae21a181a52
Sep 11 14:00:24.833058 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:24.833018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw" event={"ID":"c48d1677-1c7f-412c-a532-f870ebb388a2","Type":"ContainerStarted","Data":"d743f651108141d13d099b6d11022182387ffd4bee54c29f01106ae21a181a52"}
Sep 11 14:00:29.858232 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:29.858196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw" event={"ID":"c48d1677-1c7f-412c-a532-f870ebb388a2","Type":"ContainerStarted","Data":"1f7c37cd1c28f7964bbd473c2d11a4ab7babb5832480d5298a0263a23fbca4de"}
Sep 11 14:00:29.876365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:00:29.876312    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-j6mbw" podStartSLOduration=1.9615378049999999 podStartE2EDuration="6.876296294s" totalImagesPullingTime="4.914758489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:00:29.875196231 +0000 UTC m=+478.242785675" watchObservedRunningTime="2026-09-11 14:00:29.876296294 +0000 UTC m=+478.243885736"
Sep 11 14:01:17.931223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:17.931185    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:01:17.934328 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:17.934304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:17.936772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:17.936752    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 14:01:17.951229 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:17.951199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:01:17.979635 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:17.979605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:01:18.069440 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.069397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwsg\" (UniqueName: \"kubernetes.io/projected/d7f8fa63-feac-4144-b598-f7bff621a5d7-kube-api-access-2lwsg\") pod \"limitador-limitador-84994f7ffc-btf8f\" (UID: \"d7f8fa63-feac-4144-b598-f7bff621a5d7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.069591 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.069456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7f8fa63-feac-4144-b598-f7bff621a5d7-config-file\") pod \"limitador-limitador-84994f7ffc-btf8f\" (UID: \"d7f8fa63-feac-4144-b598-f7bff621a5d7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.170510 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.170478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7f8fa63-feac-4144-b598-f7bff621a5d7-config-file\") pod \"limitador-limitador-84994f7ffc-btf8f\" (UID: \"d7f8fa63-feac-4144-b598-f7bff621a5d7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.170683 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.170543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwsg\" (UniqueName: \"kubernetes.io/projected/d7f8fa63-feac-4144-b598-f7bff621a5d7-kube-api-access-2lwsg\") pod \"limitador-limitador-84994f7ffc-btf8f\" (UID: \"d7f8fa63-feac-4144-b598-f7bff621a5d7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.171180 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.171161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d7f8fa63-feac-4144-b598-f7bff621a5d7-config-file\") pod \"limitador-limitador-84994f7ffc-btf8f\" (UID: \"d7f8fa63-feac-4144-b598-f7bff621a5d7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.180260 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.180239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwsg\" (UniqueName: \"kubernetes.io/projected/d7f8fa63-feac-4144-b598-f7bff621a5d7-kube-api-access-2lwsg\") pod \"limitador-limitador-84994f7ffc-btf8f\" (UID: \"d7f8fa63-feac-4144-b598-f7bff621a5d7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.244233 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.244154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:18.251733 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.251708    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k2d67"]
Sep 11 14:01:18.254817 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.254801    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:18.257741 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.257720    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x7jk6\""
Sep 11 14:01:18.265434 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.265407    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k2d67"]
Sep 11 14:01:18.371990 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.371959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/7067a698-c0d4-4d64-9445-d7f0b3861154-kube-api-access-vv6qw\") pod \"authorino-7bcfd8f9cd-k2d67\" (UID: \"7067a698-c0d4-4d64-9445-d7f0b3861154\") " pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:18.472348 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.472312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/7067a698-c0d4-4d64-9445-d7f0b3861154-kube-api-access-vv6qw\") pod \"authorino-7bcfd8f9cd-k2d67\" (UID: \"7067a698-c0d4-4d64-9445-d7f0b3861154\") " pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:18.480773 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.480747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/7067a698-c0d4-4d64-9445-d7f0b3861154-kube-api-access-vv6qw\") pod \"authorino-7bcfd8f9cd-k2d67\" (UID: \"7067a698-c0d4-4d64-9445-d7f0b3861154\") " pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:18.516533 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.516461    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:01:18.519470 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:01:18.519441    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f8fa63_feac_4144_b598_f7bff621a5d7.slice/crio-94a2cee7de66da404ce2e2cb8737f9acaf9fb856a4f35f0f183aad911423ee4d WatchSource:0}: Error finding container 94a2cee7de66da404ce2e2cb8737f9acaf9fb856a4f35f0f183aad911423ee4d: Status 404 returned error can't find the container with id 94a2cee7de66da404ce2e2cb8737f9acaf9fb856a4f35f0f183aad911423ee4d
Sep 11 14:01:18.573463 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.573433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:18.839514 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:18.839478    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k2d67"]
Sep 11 14:01:18.842453 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:01:18.842424    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7067a698_c0d4_4d64_9445_d7f0b3861154.slice/crio-26a8d53a74c230eb3a7b85b34f53e41b665d473c920786761937f18fc47180b2 WatchSource:0}: Error finding container 26a8d53a74c230eb3a7b85b34f53e41b665d473c920786761937f18fc47180b2: Status 404 returned error can't find the container with id 26a8d53a74c230eb3a7b85b34f53e41b665d473c920786761937f18fc47180b2
Sep 11 14:01:19.027386 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:19.027350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67" event={"ID":"7067a698-c0d4-4d64-9445-d7f0b3861154","Type":"ContainerStarted","Data":"26a8d53a74c230eb3a7b85b34f53e41b665d473c920786761937f18fc47180b2"}
Sep 11 14:01:19.028351 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:19.028328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f" event={"ID":"d7f8fa63-feac-4144-b598-f7bff621a5d7","Type":"ContainerStarted","Data":"94a2cee7de66da404ce2e2cb8737f9acaf9fb856a4f35f0f183aad911423ee4d"}
Sep 11 14:01:21.555928 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:21.555875    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k2d67"]
Sep 11 14:01:22.042419 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.042383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67" event={"ID":"7067a698-c0d4-4d64-9445-d7f0b3861154","Type":"ContainerStarted","Data":"e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405"}
Sep 11 14:01:22.042616 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.042422    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67" podUID="7067a698-c0d4-4d64-9445-d7f0b3861154" containerName="authorino" containerID="cri-o://e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405" gracePeriod=30
Sep 11 14:01:22.044047 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.044019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f" event={"ID":"d7f8fa63-feac-4144-b598-f7bff621a5d7","Type":"ContainerStarted","Data":"a0d7f6cf616abbe04a49705151cfc65773a0e342c637114e53831205e9360615"}
Sep 11 14:01:22.044202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.044184    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:22.060548 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.060471    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67" podStartSLOduration=1.137739702 podStartE2EDuration="4.060454678s" totalImagesPullingTime="2.922714976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:22.060213877 +0000 UTC m=+530.427803320" watchObservedRunningTime="2026-09-11 14:01:22.060454678 +0000 UTC m=+530.428044185"
Sep 11 14:01:22.081303 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.081250    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f" podStartSLOduration=1.841781413 podStartE2EDuration="5.081234698s" totalImagesPullingTime="3.239453285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:22.080312153 +0000 UTC m=+530.447901597" watchObservedRunningTime="2026-09-11 14:01:22.081234698 +0000 UTC m=+530.448824140"
Sep 11 14:01:22.276752 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.276722    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:22.407672 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.407587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/7067a698-c0d4-4d64-9445-d7f0b3861154-kube-api-access-vv6qw\") pod \"7067a698-c0d4-4d64-9445-d7f0b3861154\" (UID: \"7067a698-c0d4-4d64-9445-d7f0b3861154\") "
Sep 11 14:01:22.409826 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.409806    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7067a698-c0d4-4d64-9445-d7f0b3861154-kube-api-access-vv6qw" (OuterVolumeSpecName: "kube-api-access-vv6qw") pod "7067a698-c0d4-4d64-9445-d7f0b3861154" (UID: "7067a698-c0d4-4d64-9445-d7f0b3861154"). InnerVolumeSpecName "kube-api-access-vv6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:01:22.508220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:22.508172    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6qw\" (UniqueName: \"kubernetes.io/projected/7067a698-c0d4-4d64-9445-d7f0b3861154-kube-api-access-vv6qw\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:01:23.048959 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.048854    2561 generic.go:356] "Generic (PLEG): container finished" podID="7067a698-c0d4-4d64-9445-d7f0b3861154" containerID="e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405" exitCode=0
Sep 11 14:01:23.048959 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.048903    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67"
Sep 11 14:01:23.048959 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.048940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67" event={"ID":"7067a698-c0d4-4d64-9445-d7f0b3861154","Type":"ContainerDied","Data":"e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405"}
Sep 11 14:01:23.049534 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.048983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-k2d67" event={"ID":"7067a698-c0d4-4d64-9445-d7f0b3861154","Type":"ContainerDied","Data":"26a8d53a74c230eb3a7b85b34f53e41b665d473c920786761937f18fc47180b2"}
Sep 11 14:01:23.049534 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.049001    2561 scope.go:117] "RemoveContainer" containerID="e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405"
Sep 11 14:01:23.062278 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.062261    2561 scope.go:117] "RemoveContainer" containerID="e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405"
Sep 11 14:01:23.062553 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:01:23.062534    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405\": container with ID starting with e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405 not found: ID does not exist" containerID="e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405"
Sep 11 14:01:23.062599 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.062561    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405"} err="failed to get container status \"e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405\": rpc error: code = NotFound desc = could not find container \"e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405\": container with ID starting with e618d1b7ab667f2e292ad5e83143f510efc5c1a401b67f2a00affdbc4e19f405 not found: ID does not exist"
Sep 11 14:01:23.072362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.072340    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k2d67"]
Sep 11 14:01:23.078152 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:23.078132    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-k2d67"]
Sep 11 14:01:24.116371 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:24.116339    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7067a698-c0d4-4d64-9445-d7f0b3861154" path="/var/lib/kubelet/pods/7067a698-c0d4-4d64-9445-d7f0b3861154/volumes"
Sep 11 14:01:33.049864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:33.049834    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-btf8f"
Sep 11 14:01:50.503363 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.503327    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7ff745f5f8-8v4lq"]
Sep 11 14:01:50.503835 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.503587    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7067a698-c0d4-4d64-9445-d7f0b3861154" containerName="authorino"
Sep 11 14:01:50.503835 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.503599    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7067a698-c0d4-4d64-9445-d7f0b3861154" containerName="authorino"
Sep 11 14:01:50.503835 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.503665    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7067a698-c0d4-4d64-9445-d7f0b3861154" containerName="authorino"
Sep 11 14:01:50.509260 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.509240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.513534 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.513513    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x7jk6\""
Sep 11 14:01:50.514724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.514701    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 14:01:50.526099 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.526072    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff745f5f8-8v4lq"]
Sep 11 14:01:50.617409 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.617370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4f4v\" (UniqueName: \"kubernetes.io/projected/643d40fa-10a0-45bf-9c04-551ed5285488-kube-api-access-t4f4v\") pod \"authorino-7ff745f5f8-8v4lq\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") " pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.617584 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.617471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/643d40fa-10a0-45bf-9c04-551ed5285488-tls-cert\") pod \"authorino-7ff745f5f8-8v4lq\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") " pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.718776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.718736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4f4v\" (UniqueName: \"kubernetes.io/projected/643d40fa-10a0-45bf-9c04-551ed5285488-kube-api-access-t4f4v\") pod \"authorino-7ff745f5f8-8v4lq\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") " pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.718968 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.718814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/643d40fa-10a0-45bf-9c04-551ed5285488-tls-cert\") pod \"authorino-7ff745f5f8-8v4lq\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") " pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.721226 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.721201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/643d40fa-10a0-45bf-9c04-551ed5285488-tls-cert\") pod \"authorino-7ff745f5f8-8v4lq\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") " pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.727398 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.727375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4f4v\" (UniqueName: \"kubernetes.io/projected/643d40fa-10a0-45bf-9c04-551ed5285488-kube-api-access-t4f4v\") pod \"authorino-7ff745f5f8-8v4lq\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") " pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:50.818107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:50.818074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:01:51.085857 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:51.085763    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff745f5f8-8v4lq"]
Sep 11 14:01:51.089208 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:01:51.089177    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod643d40fa_10a0_45bf_9c04_551ed5285488.slice/crio-2df2e2f10b2951573c92fd03aec48743e21f0bc31a605f25311e7b21d16855cb WatchSource:0}: Error finding container 2df2e2f10b2951573c92fd03aec48743e21f0bc31a605f25311e7b21d16855cb: Status 404 returned error can't find the container with id 2df2e2f10b2951573c92fd03aec48743e21f0bc31a605f25311e7b21d16855cb
Sep 11 14:01:51.149320 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:51.149279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq" event={"ID":"643d40fa-10a0-45bf-9c04-551ed5285488","Type":"ContainerStarted","Data":"2df2e2f10b2951573c92fd03aec48743e21f0bc31a605f25311e7b21d16855cb"}
Sep 11 14:01:52.153720 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:52.153632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq" event={"ID":"643d40fa-10a0-45bf-9c04-551ed5285488","Type":"ContainerStarted","Data":"d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0"}
Sep 11 14:01:52.171728 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:01:52.171678    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq" podStartSLOduration=1.447646521 podStartE2EDuration="2.171660749s" totalImagesPullingTime="724.014228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:52.171198646 +0000 UTC m=+560.538788101" watchObservedRunningTime="2026-09-11 14:01:52.171660749 +0000 UTC m=+560.539250193"
Sep 11 14:02:32.088952 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:02:32.088925    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:02:32.090213 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:02:32.090192    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:02:35.524279 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:02:35.524235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:02:56.893171 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:02:56.893077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:03:44.400722 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:03:44.400674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:03:56.627461 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:03:56.627422    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:04:05.986277 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:05.986238    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:04:15.610018 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:15.609983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:04:29.794118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:29.794028    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:04:48.279924 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.279884    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"]
Sep 11 14:04:48.283037 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.283019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.295741 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.295718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4r8ml\""
Sep 11 14:04:48.295876 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.295745    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 14:04:48.295876 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.295769    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 14:04:48.296013 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.296004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 14:04:48.319465 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.319438    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"]
Sep 11 14:04:48.383481 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.383442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.383653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.383489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.383653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.383572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.383653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.383619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.383653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.383646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.383807 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.383688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wqw\" (UniqueName: \"kubernetes.io/projected/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-kube-api-access-g5wqw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.484674 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.484637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.484674 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.484676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.484965 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.484717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wqw\" (UniqueName: \"kubernetes.io/projected/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-kube-api-access-g5wqw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.484965 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.484762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.484965 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.484790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.484965 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.484836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.485191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.485159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.485191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.485178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.485267 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.485241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.487066 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.487049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.487332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.487315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.494142 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.494116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wqw\" (UniqueName: \"kubernetes.io/projected/866f5d39-dc4e-4a9b-b0ce-559b54cf2c69-kube-api-access-g5wqw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g\" (UID: \"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.591239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.591193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:48.861221 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.861189    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"]
Sep 11 14:04:48.863681 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:04:48.863649    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod866f5d39_dc4e_4a9b_b0ce_559b54cf2c69.slice/crio-284bf62e41a09dc4d15cf7c6d60e3071946b2d4409ea733188f4129d42ca221d WatchSource:0}: Error finding container 284bf62e41a09dc4d15cf7c6d60e3071946b2d4409ea733188f4129d42ca221d: Status 404 returned error can't find the container with id 284bf62e41a09dc4d15cf7c6d60e3071946b2d4409ea733188f4129d42ca221d
Sep 11 14:04:48.864932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:48.864902    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:04:49.795950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:49.795895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g" event={"ID":"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69","Type":"ContainerStarted","Data":"284bf62e41a09dc4d15cf7c6d60e3071946b2d4409ea733188f4129d42ca221d"}
Sep 11 14:04:50.590864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:50.590822    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:04:50.802741 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:50.802707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g" event={"ID":"866f5d39-dc4e-4a9b-b0ce-559b54cf2c69","Type":"ContainerStarted","Data":"8fc0b6484ec5dbea24276897d7139fe4344f47f78e763efcafe21403e64f0e6e"}
Sep 11 14:04:50.822969 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:50.822899    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g" podStartSLOduration=1.830616562 podStartE2EDuration="2.822879069s" totalImagesPullingTime="992.262507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:04:50.822291067 +0000 UTC m=+739.189880511" watchObservedRunningTime="2026-09-11 14:04:50.822879069 +0000 UTC m=+739.190468511"
Sep 11 14:04:58.591930 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:58.591869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:58.592518 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:58.591950    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:58.604244 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:58.604220    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:04:58.844849 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:04:58.844770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g"
Sep 11 14:06:08.085825 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.085794    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c9b89ddf5-jbscj"]
Sep 11 14:06:08.088986 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.088967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.104745 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.104723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9b89ddf5-jbscj"]
Sep 11 14:06:08.228593 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.228558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgcsw\" (UniqueName: \"kubernetes.io/projected/05bbb342-e3de-4680-bc6d-b004e33489df-kube-api-access-jgcsw\") pod \"authorino-6c9b89ddf5-jbscj\" (UID: \"05bbb342-e3de-4680-bc6d-b004e33489df\") " pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.228759 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.228625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05bbb342-e3de-4680-bc6d-b004e33489df-tls-cert\") pod \"authorino-6c9b89ddf5-jbscj\" (UID: \"05bbb342-e3de-4680-bc6d-b004e33489df\") " pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.329628 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.329592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05bbb342-e3de-4680-bc6d-b004e33489df-tls-cert\") pod \"authorino-6c9b89ddf5-jbscj\" (UID: \"05bbb342-e3de-4680-bc6d-b004e33489df\") " pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.329780 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.329644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgcsw\" (UniqueName: \"kubernetes.io/projected/05bbb342-e3de-4680-bc6d-b004e33489df-kube-api-access-jgcsw\") pod \"authorino-6c9b89ddf5-jbscj\" (UID: \"05bbb342-e3de-4680-bc6d-b004e33489df\") " pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.331805 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.331786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05bbb342-e3de-4680-bc6d-b004e33489df-tls-cert\") pod \"authorino-6c9b89ddf5-jbscj\" (UID: \"05bbb342-e3de-4680-bc6d-b004e33489df\") " pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.344027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.343965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgcsw\" (UniqueName: \"kubernetes.io/projected/05bbb342-e3de-4680-bc6d-b004e33489df-kube-api-access-jgcsw\") pod \"authorino-6c9b89ddf5-jbscj\" (UID: \"05bbb342-e3de-4680-bc6d-b004e33489df\") " pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.397612 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.397576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9b89ddf5-jbscj"
Sep 11 14:06:08.676134 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:08.676109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9b89ddf5-jbscj"]
Sep 11 14:06:08.678625 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:08.678595    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05bbb342_e3de_4680_bc6d_b004e33489df.slice/crio-2cf41c0fe80897d875f3f22a362512dc785096c95719ca66a79016b6fd842ff0 WatchSource:0}: Error finding container 2cf41c0fe80897d875f3f22a362512dc785096c95719ca66a79016b6fd842ff0: Status 404 returned error can't find the container with id 2cf41c0fe80897d875f3f22a362512dc785096c95719ca66a79016b6fd842ff0
Sep 11 14:06:09.094177 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:09.094143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9b89ddf5-jbscj" event={"ID":"05bbb342-e3de-4680-bc6d-b004e33489df","Type":"ContainerStarted","Data":"2cf41c0fe80897d875f3f22a362512dc785096c95719ca66a79016b6fd842ff0"}
Sep 11 14:06:10.100561 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.100515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9b89ddf5-jbscj" event={"ID":"05bbb342-e3de-4680-bc6d-b004e33489df","Type":"ContainerStarted","Data":"bb84e7124653ae55fca6ea97bea1b36643def1aa4f8a71fe484d99061908dbdc"}
Sep 11 14:06:10.124290 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.124233    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c9b89ddf5-jbscj" podStartSLOduration=1.020228233 podStartE2EDuration="2.124216775s" totalImagesPullingTime="1.103988542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:10.122182988 +0000 UTC m=+818.489772455" watchObservedRunningTime="2026-09-11 14:06:10.124216775 +0000 UTC m=+818.491806219"
Sep 11 14:06:10.230819 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.230779    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ff745f5f8-8v4lq"]
Sep 11 14:06:10.231072 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.231028    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq" podUID="643d40fa-10a0-45bf-9c04-551ed5285488" containerName="authorino" containerID="cri-o://d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0" gracePeriod=30
Sep 11 14:06:10.475450 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.475420    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:06:10.649538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.649394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4f4v\" (UniqueName: \"kubernetes.io/projected/643d40fa-10a0-45bf-9c04-551ed5285488-kube-api-access-t4f4v\") pod \"643d40fa-10a0-45bf-9c04-551ed5285488\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") "
Sep 11 14:06:10.649538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.649470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/643d40fa-10a0-45bf-9c04-551ed5285488-tls-cert\") pod \"643d40fa-10a0-45bf-9c04-551ed5285488\" (UID: \"643d40fa-10a0-45bf-9c04-551ed5285488\") "
Sep 11 14:06:10.651560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.651525    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643d40fa-10a0-45bf-9c04-551ed5285488-kube-api-access-t4f4v" (OuterVolumeSpecName: "kube-api-access-t4f4v") pod "643d40fa-10a0-45bf-9c04-551ed5285488" (UID: "643d40fa-10a0-45bf-9c04-551ed5285488"). InnerVolumeSpecName "kube-api-access-t4f4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:06:10.659416 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.659384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643d40fa-10a0-45bf-9c04-551ed5285488-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "643d40fa-10a0-45bf-9c04-551ed5285488" (UID: "643d40fa-10a0-45bf-9c04-551ed5285488"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:06:10.751026 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.750995    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4f4v\" (UniqueName: \"kubernetes.io/projected/643d40fa-10a0-45bf-9c04-551ed5285488-kube-api-access-t4f4v\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:06:10.751026 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:10.751019    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/643d40fa-10a0-45bf-9c04-551ed5285488-tls-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:06:11.106364 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.106323    2561 generic.go:356] "Generic (PLEG): container finished" podID="643d40fa-10a0-45bf-9c04-551ed5285488" containerID="d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0" exitCode=0
Sep 11 14:06:11.106364 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.106367    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq"
Sep 11 14:06:11.106899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.106398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq" event={"ID":"643d40fa-10a0-45bf-9c04-551ed5285488","Type":"ContainerDied","Data":"d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0"}
Sep 11 14:06:11.106899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.106441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff745f5f8-8v4lq" event={"ID":"643d40fa-10a0-45bf-9c04-551ed5285488","Type":"ContainerDied","Data":"2df2e2f10b2951573c92fd03aec48743e21f0bc31a605f25311e7b21d16855cb"}
Sep 11 14:06:11.106899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.106459    2561 scope.go:117] "RemoveContainer" containerID="d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0"
Sep 11 14:06:11.122096 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.122064    2561 scope.go:117] "RemoveContainer" containerID="d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0"
Sep 11 14:06:11.122450 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:11.122430    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0\": container with ID starting with d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0 not found: ID does not exist" containerID="d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0"
Sep 11 14:06:11.122517 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.122459    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0"} err="failed to get container status \"d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0\": rpc error: code = NotFound desc = could not find container \"d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0\": container with ID starting with d1b7586669b02bd69e9e936e99ee24cacb145ef7779634906c5a842e090ca5c0 not found: ID does not exist"
Sep 11 14:06:11.172997 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.172964    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ff745f5f8-8v4lq"]
Sep 11 14:06:11.185330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:11.185300    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7ff745f5f8-8v4lq"]
Sep 11 14:06:12.115459 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:12.115427    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643d40fa-10a0-45bf-9c04-551ed5285488" path="/var/lib/kubelet/pods/643d40fa-10a0-45bf-9c04-551ed5285488/volumes"
Sep 11 14:06:30.670049 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.670012    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-96ba06"]
Sep 11 14:06:30.670439 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.670289    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643d40fa-10a0-45bf-9c04-551ed5285488" containerName="authorino"
Sep 11 14:06:30.670439 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.670300    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="643d40fa-10a0-45bf-9c04-551ed5285488" containerName="authorino"
Sep 11 14:06:30.670439 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.670358    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="643d40fa-10a0-45bf-9c04-551ed5285488" containerName="authorino"
Sep 11 14:06:30.673156 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.673138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:30.678216 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.678186    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m8d75\""
Sep 11 14:06:30.689772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.689737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-96ba06"]
Sep 11 14:06:30.694665 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.694639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn5s\" (UniqueName: \"kubernetes.io/projected/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6-kube-api-access-dpn5s\") pod \"test-curl-7755-96ba06\" (UID: \"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6\") " pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:30.795321 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.795279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn5s\" (UniqueName: \"kubernetes.io/projected/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6-kube-api-access-dpn5s\") pod \"test-curl-7755-96ba06\" (UID: \"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6\") " pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:30.829220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.829193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn5s\" (UniqueName: \"kubernetes.io/projected/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6-kube-api-access-dpn5s\") pod \"test-curl-7755-96ba06\" (UID: \"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6\") " pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:30.981465 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:30.981365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:31.314835 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:31.314812    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-96ba06"]
Sep 11 14:06:31.316654 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:31.316627    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddda9e1cb_2084_4c1f_b43b_a91d1eef74b6.slice/crio-ce8c9c6985ea91ff5132a54b15fd67dd319aacfb69fd31b5260f43ee4284c548 WatchSource:0}: Error finding container ce8c9c6985ea91ff5132a54b15fd67dd319aacfb69fd31b5260f43ee4284c548: Status 404 returned error can't find the container with id ce8c9c6985ea91ff5132a54b15fd67dd319aacfb69fd31b5260f43ee4284c548
Sep 11 14:06:32.193788 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:32.193747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-96ba06" event={"ID":"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6","Type":"ContainerStarted","Data":"ce8c9c6985ea91ff5132a54b15fd67dd319aacfb69fd31b5260f43ee4284c548"}
Sep 11 14:06:33.295790 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:33.295747    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:06:34.203851 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:34.203814    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-96ba06" event={"ID":"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6","Type":"ContainerStarted","Data":"bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876"}
Sep 11 14:06:34.365055 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:34.364996    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-96ba06" podStartSLOduration=2.168767566 podStartE2EDuration="4.36497707s" totalImagesPullingTime="2.196209504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:34.291077983 +0000 UTC m=+842.658667425" watchObservedRunningTime="2026-09-11 14:06:34.36497707 +0000 UTC m=+842.732566512"
Sep 11 14:06:35.430774 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.430686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-96ba06"]
Sep 11 14:06:35.453881 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.453846    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-96ba06"]
Sep 11 14:06:35.642089 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.642056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-80c08d"]
Sep 11 14:06:35.642326 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.642313    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" containerName="test-curl-7755-96ba06"
Sep 11 14:06:35.642387 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.642328    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" containerName="test-curl-7755-96ba06"
Sep 11 14:06:35.642387 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.642385    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" containerName="test-curl-7755-96ba06"
Sep 11 14:06:35.645316 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.645301    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:35.669787 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.669759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-80c08d"]
Sep 11 14:06:35.734990 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.734866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvz9g\" (UniqueName: \"kubernetes.io/projected/cfe8f9ac-9d71-4c59-b074-ab882eada946-kube-api-access-gvz9g\") pod \"test-curl-7755-80c08d\" (UID: \"cfe8f9ac-9d71-4c59-b074-ab882eada946\") " pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:35.835284 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.835245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvz9g\" (UniqueName: \"kubernetes.io/projected/cfe8f9ac-9d71-4c59-b074-ab882eada946-kube-api-access-gvz9g\") pod \"test-curl-7755-80c08d\" (UID: \"cfe8f9ac-9d71-4c59-b074-ab882eada946\") " pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:35.853887 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.853856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvz9g\" (UniqueName: \"kubernetes.io/projected/cfe8f9ac-9d71-4c59-b074-ab882eada946-kube-api-access-gvz9g\") pod \"test-curl-7755-80c08d\" (UID: \"cfe8f9ac-9d71-4c59-b074-ab882eada946\") " pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:35.953814 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:35.953772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:36.212764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:36.212722    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-96ba06" podUID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" containerName="test-curl-7755-96ba06" containerID="cri-o://bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876" gracePeriod=2
Sep 11 14:06:36.242275 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:36.242247    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfe8f9ac_9d71_4c59_b074_ab882eada946.slice/crio-d131439eeb495abb34f8f5893642aa143c685111072ce8c21efd83c566b4d8f5 WatchSource:0}: Error finding container d131439eeb495abb34f8f5893642aa143c685111072ce8c21efd83c566b4d8f5: Status 404 returned error can't find the container with id d131439eeb495abb34f8f5893642aa143c685111072ce8c21efd83c566b4d8f5
Sep 11 14:06:36.245176 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:36.245148    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-80c08d"]
Sep 11 14:06:37.219585 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.219553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-80c08d" event={"ID":"cfe8f9ac-9d71-4c59-b074-ab882eada946","Type":"ContainerStarted","Data":"ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078"}
Sep 11 14:06:37.219955 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.219595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-80c08d" event={"ID":"cfe8f9ac-9d71-4c59-b074-ab882eada946","Type":"ContainerStarted","Data":"d131439eeb495abb34f8f5893642aa143c685111072ce8c21efd83c566b4d8f5"}
Sep 11 14:06:37.263281 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.263226    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-80c08d" podStartSLOduration=2.263208183 podStartE2EDuration="2.263208183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:37.255158602 +0000 UTC m=+845.622748045" watchObservedRunningTime="2026-09-11 14:06:37.263208183 +0000 UTC m=+845.630797626"
Sep 11 14:06:37.552354 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.552276    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"]
Sep 11 14:06:37.555590 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.555569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.558895 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.558870    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-abfca2\""
Sep 11 14:06:37.559827 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.559811    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-abfca2-dockercfg-fdlrk\""
Sep 11 14:06:37.569451 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.569428    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"]
Sep 11 14:06:37.649332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.649298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5cq\" (UniqueName: \"kubernetes.io/projected/54feb8e7-d36f-47d3-97ea-c053f86909d9-kube-api-access-kx5cq\") pod \"payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.649489 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.649360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54feb8e7-d36f-47d3-97ea-c053f86909d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.750663 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.750619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5cq\" (UniqueName: \"kubernetes.io/projected/54feb8e7-d36f-47d3-97ea-c053f86909d9-kube-api-access-kx5cq\") pod \"payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.750803 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.750694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54feb8e7-d36f-47d3-97ea-c053f86909d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.751497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.751476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54feb8e7-d36f-47d3-97ea-c053f86909d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.763281 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.763255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5cq\" (UniqueName: \"kubernetes.io/projected/54feb8e7-d36f-47d3-97ea-c053f86909d9-kube-api-access-kx5cq\") pod \"payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:37.865946 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:37.865891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:38.159203 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.159174    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"]
Sep 11 14:06:38.160945 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:38.160892    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54feb8e7_d36f_47d3_97ea_c053f86909d9.slice/crio-76676a7eb3fdedec6e4ffecedec91c0656813ca07bcf16bcaf2e7e3e9dc810b0 WatchSource:0}: Error finding container 76676a7eb3fdedec6e4ffecedec91c0656813ca07bcf16bcaf2e7e3e9dc810b0: Status 404 returned error can't find the container with id 76676a7eb3fdedec6e4ffecedec91c0656813ca07bcf16bcaf2e7e3e9dc810b0
Sep 11 14:06:38.197486 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.197455    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-80c08d"]
Sep 11 14:06:38.199241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.199217    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-80c08d"]
Sep 11 14:06:38.227059 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.227026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh" event={"ID":"54feb8e7-d36f-47d3-97ea-c053f86909d9","Type":"ContainerStarted","Data":"76676a7eb3fdedec6e4ffecedec91c0656813ca07bcf16bcaf2e7e3e9dc810b0"}
Sep 11 14:06:38.227430 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:38.227285    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7755-80c08d" container="test-curl-7755-80c08d"
Sep 11 14:06:38.369986 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.369962    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:38.455973 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.455857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpn5s\" (UniqueName: \"kubernetes.io/projected/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6-kube-api-access-dpn5s\") pod \"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6\" (UID: \"dda9e1cb-2084-4c1f-b43b-a91d1eef74b6\") "
Sep 11 14:06:38.458053 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.458026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6-kube-api-access-dpn5s" (OuterVolumeSpecName: "kube-api-access-dpn5s") pod "dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" (UID: "dda9e1cb-2084-4c1f-b43b-a91d1eef74b6"). InnerVolumeSpecName "kube-api-access-dpn5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:06:38.556921 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:38.556881    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn5s\" (UniqueName: \"kubernetes.io/projected/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6-kube-api-access-dpn5s\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:06:39.232374 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:39.232333    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" containerID="bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876" exitCode=137
Sep 11 14:06:39.232374 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:39.232374    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:39.232999 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:39.232442    2561 scope.go:117] "RemoveContainer" containerID="bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876"
Sep 11 14:06:39.232999 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:39.232658    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-80c08d" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" containerName="test-curl-7755-80c08d" containerID="cri-o://ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078" gracePeriod=2
Sep 11 14:06:39.240229 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:39.240211    2561 scope.go:117] "RemoveContainer" containerID="bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876"
Sep 11 14:06:39.240669 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:39.240546    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876\": container with ID starting with bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876 not found: ID does not exist" containerID="bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876"
Sep 11 14:06:39.240669 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:39.240583    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876"} err="failed to get container status \"bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876\": rpc error: code = NotFound desc = could not find container \"bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876\": container with ID starting with bcdbdb5b0081cff2c96f4ff721655d9ef2d3ad7fd4de7906d1001005f2580876 not found: ID does not exist"
Sep 11 14:06:39.248987 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:39.248949    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-80c08d\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:39.260275 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:39.260232    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-96ba06\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" pod="opendatahub/test-curl-7755-96ba06"
Sep 11 14:06:39.263792 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:39.263768    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-80c08d\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:40.117052 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:40.117003    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda9e1cb-2084-4c1f-b43b-a91d1eef74b6" path="/var/lib/kubelet/pods/dda9e1cb-2084-4c1f-b43b-a91d1eef74b6/volumes"
Sep 11 14:06:40.238609 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:40.238514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh" event={"ID":"54feb8e7-d36f-47d3-97ea-c053f86909d9","Type":"ContainerStarted","Data":"c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98"}
Sep 11 14:06:40.239078 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:40.238654    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:41.435090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:41.435064    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:41.482875 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:41.482836    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvz9g\" (UniqueName: \"kubernetes.io/projected/cfe8f9ac-9d71-4c59-b074-ab882eada946-kube-api-access-gvz9g\") pod \"cfe8f9ac-9d71-4c59-b074-ab882eada946\" (UID: \"cfe8f9ac-9d71-4c59-b074-ab882eada946\") "
Sep 11 14:06:41.485396 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:41.485355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe8f9ac-9d71-4c59-b074-ab882eada946-kube-api-access-gvz9g" (OuterVolumeSpecName: "kube-api-access-gvz9g") pod "cfe8f9ac-9d71-4c59-b074-ab882eada946" (UID: "cfe8f9ac-9d71-4c59-b074-ab882eada946"). InnerVolumeSpecName "kube-api-access-gvz9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:06:41.583789 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:41.583757    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvz9g\" (UniqueName: \"kubernetes.io/projected/cfe8f9ac-9d71-4c59-b074-ab882eada946-kube-api-access-gvz9g\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:06:42.116297 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:42.116264    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" path="/var/lib/kubelet/pods/cfe8f9ac-9d71-4c59-b074-ab882eada946/volumes"
Sep 11 14:06:42.248377 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:42.248337    2561 generic.go:356] "Generic (PLEG): container finished" podID="cfe8f9ac-9d71-4c59-b074-ab882eada946" containerID="ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078" exitCode=137
Sep 11 14:06:42.248377 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:42.248376    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-80c08d"
Sep 11 14:06:42.248620 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:42.248427    2561 scope.go:117] "RemoveContainer" containerID="ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078"
Sep 11 14:06:42.257008 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:42.256985    2561 scope.go:117] "RemoveContainer" containerID="ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078"
Sep 11 14:06:42.257290 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:06:42.257267    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078\": container with ID starting with ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078 not found: ID does not exist" containerID="ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078"
Sep 11 14:06:42.257344 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:42.257299    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078"} err="failed to get container status \"ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078\": rpc error: code = NotFound desc = could not find container \"ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078\": container with ID starting with ec02a1864a50d809707fa093b01b8dc9582da39e4aed1ef7a25dd1428a2f3078 not found: ID does not exist"
Sep 11 14:06:45.957125 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.957063    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh" podStartSLOduration=7.244998943 podStartE2EDuration="8.957042489s" totalImagesPullingTime="1.712043546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:40.586977083 +0000 UTC m=+848.954566526" watchObservedRunningTime="2026-09-11 14:06:45.957042489 +0000 UTC m=+854.324631935"
Sep 11 14:06:45.957681 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.957664    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"]
Sep 11 14:06:45.957961 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.957949    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" containerName="test-curl-7755-80c08d"
Sep 11 14:06:45.958005 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.957963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" containerName="test-curl-7755-80c08d"
Sep 11 14:06:45.958039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.958026    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe8f9ac-9d71-4c59-b074-ab882eada946" containerName="test-curl-7755-80c08d"
Sep 11 14:06:45.962117 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.962103    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:45.966046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.966026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4e6230ad-openshift-default-dockercfg-l2q5b\""
Sep 11 14:06:45.966541 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:45.966524    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:06:46.015853 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.015818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-istio-token\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.015853 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.015854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kc28\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-kube-api-access-6kc28\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.015888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-certs\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.015966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-socket\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.015995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872f71a3-fac8-463b-928e-76d5e22c0636-istiod-ca-cert\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.016022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-data\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.016096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-credential-socket\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.016122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-envoy\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.016325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.016173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872f71a3-fac8-463b-928e-76d5e22c0636-istio-podinfo\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.022158 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.022133    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"]
Sep 11 14:06:46.117556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-credential-socket\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.117747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-envoy\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.117747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872f71a3-fac8-463b-928e-76d5e22c0636-istio-podinfo\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.117747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-istio-token\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.117747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc28\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-kube-api-access-6kc28\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.117747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-certs\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-socket\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872f71a3-fac8-463b-928e-76d5e22c0636-istiod-ca-cert\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-data\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.117979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-credential-socket\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118224 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.118170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-data\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.118270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-certs\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.118433 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.118408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-socket\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.119074 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.119049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872f71a3-fac8-463b-928e-76d5e22c0636-istiod-ca-cert\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.120077 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.120052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-envoy\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.120295 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.120274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872f71a3-fac8-463b-928e-76d5e22c0636-istio-podinfo\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.146040 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.146016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kc28\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-kube-api-access-6kc28\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.149723 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.149701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-istio-token\") pod \"e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") " pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.271670 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.271589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:46.570257 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:46.570227    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"]
Sep 11 14:06:46.573460 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:46.573431    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872f71a3_fac8_463b_928e_76d5e22c0636.slice/crio-c4f14168283b33fba86e4d829e2f27596c549b5fa55497debae02302cee2920c WatchSource:0}: Error finding container c4f14168283b33fba86e4d829e2f27596c549b5fa55497debae02302cee2920c: Status 404 returned error can't find the container with id c4f14168283b33fba86e4d829e2f27596c549b5fa55497debae02302cee2920c
Sep 11 14:06:47.273288 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.273251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" event={"ID":"872f71a3-fac8-463b-928e-76d5e22c0636","Type":"ContainerStarted","Data":"c4f14168283b33fba86e4d829e2f27596c549b5fa55497debae02302cee2920c"}
Sep 11 14:06:47.302032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.301987    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"]
Sep 11 14:06:47.305295 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.305260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.314732 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.314712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-0bd77b-dockercfg-98v24\""
Sep 11 14:06:47.314871 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.314792    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0bd77b\""
Sep 11 14:06:47.327697 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.327673    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"]
Sep 11 14:06:47.430007 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.429970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzxp\" (UniqueName: \"kubernetes.io/projected/66828599-9e10-4384-be49-a72e1cf11caa-kube-api-access-ctzxp\") pod \"payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") " pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.430198 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.430083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66828599-9e10-4384-be49-a72e1cf11caa-plugins-config-volume\") pod \"payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") " pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.530665 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.530579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66828599-9e10-4384-be49-a72e1cf11caa-plugins-config-volume\") pod \"payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") " pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.530807 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.530683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzxp\" (UniqueName: \"kubernetes.io/projected/66828599-9e10-4384-be49-a72e1cf11caa-kube-api-access-ctzxp\") pod \"payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") " pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.531376 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.531356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66828599-9e10-4384-be49-a72e1cf11caa-plugins-config-volume\") pod \"payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") " pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.546433 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.546405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzxp\" (UniqueName: \"kubernetes.io/projected/66828599-9e10-4384-be49-a72e1cf11caa-kube-api-access-ctzxp\") pod \"payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") " pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.617413 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.617380    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:47.903227 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:47.903194    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"]
Sep 11 14:06:47.905944 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:47.905898    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66828599_9e10_4384_be49_a72e1cf11caa.slice/crio-80222956430b14ca3f72fb242c19c6a2a932a43bcf8179b29646e11fcc293c24 WatchSource:0}: Error finding container 80222956430b14ca3f72fb242c19c6a2a932a43bcf8179b29646e11fcc293c24: Status 404 returned error can't find the container with id 80222956430b14ca3f72fb242c19c6a2a932a43bcf8179b29646e11fcc293c24
Sep 11 14:06:48.280301 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:48.280216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g" event={"ID":"66828599-9e10-4384-be49-a72e1cf11caa","Type":"ContainerStarted","Data":"c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d"}
Sep 11 14:06:48.280301 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:48.280273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g" event={"ID":"66828599-9e10-4384-be49-a72e1cf11caa","Type":"ContainerStarted","Data":"80222956430b14ca3f72fb242c19c6a2a932a43bcf8179b29646e11fcc293c24"}
Sep 11 14:06:48.280807 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:48.280305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:48.306426 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:48.306368    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g" podStartSLOduration=1.306347098 podStartE2EDuration="1.306347098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:48.304423607 +0000 UTC m=+856.672013052" watchObservedRunningTime="2026-09-11 14:06:48.306347098 +0000 UTC m=+856.673936542"
Sep 11 14:06:48.591960 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:48.591903    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:06:49.085542 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.085509    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:06:49.085625 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.085575    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:06:49.085625 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.085601    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:06:49.287158 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.287119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" event={"ID":"872f71a3-fac8-463b-928e-76d5e22c0636","Type":"ContainerStarted","Data":"22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"}
Sep 11 14:06:49.314497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.314442    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podStartSLOduration=1.804693449 podStartE2EDuration="4.314425119s" totalImagesPullingTime="2.50973167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:49.313839156 +0000 UTC m=+857.681428603" watchObservedRunningTime="2026-09-11 14:06:49.314425119 +0000 UTC m=+857.682014562"
Sep 11 14:06:49.560771 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.560735    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"]
Sep 11 14:06:49.564375 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.564357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.567442 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.567422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-918362dd-kserve-self-signed-certs\""
Sep 11 14:06:49.572885 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.572863    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"]
Sep 11 14:06:49.649867 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.649829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddtq\" (UniqueName: \"kubernetes.io/projected/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-kube-api-access-2ddtq\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.649867 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.649870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tls-certs\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.650095 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.649890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-model-cache\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.650095 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.649934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-home\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.650095 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.649975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-dshm\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.650095 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.650018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tmp-dir\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751303 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tmp-dir\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751482 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddtq\" (UniqueName: \"kubernetes.io/projected/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-kube-api-access-2ddtq\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751482 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tls-certs\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751482 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-model-cache\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751482 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-home\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751482 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-dshm\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751729 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tmp-dir\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751782 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-model-cache\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.751837 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.751787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-home\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.753678 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.753657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-dshm\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.753962 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.753941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tls-certs\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.765929 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.765879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddtq\" (UniqueName: \"kubernetes.io/projected/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-kube-api-access-2ddtq\") pod \"e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") " pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:49.875814 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:49.875737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:50.155607 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.155431    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"]
Sep 11 14:06:50.158061 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:50.158031    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda74e6bd3_cd3a_4552_bb00_65bc829fb16b.slice/crio-a33ccb36d494695d17e0cf865338338924bbf55379507eb5222fb3ffb02b82fb WatchSource:0}: Error finding container a33ccb36d494695d17e0cf865338338924bbf55379507eb5222fb3ffb02b82fb: Status 404 returned error can't find the container with id a33ccb36d494695d17e0cf865338338924bbf55379507eb5222fb3ffb02b82fb
Sep 11 14:06:50.272584 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.272554    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:50.277849 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.277825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:50.292628 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.292589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l" event={"ID":"a74e6bd3-cd3a-4552-bb00-65bc829fb16b","Type":"ContainerStarted","Data":"6dce8a19513e8d1759e57ce33ee4fd931483daaa5a5e38109620f1ff85ae8457"}
Sep 11 14:06:50.292628 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.292633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l" event={"ID":"a74e6bd3-cd3a-4552-bb00-65bc829fb16b","Type":"ContainerStarted","Data":"a33ccb36d494695d17e0cf865338338924bbf55379507eb5222fb3ffb02b82fb"}
Sep 11 14:06:50.293141 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.292798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:50.293618 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.293599    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:06:50.362066 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:50.361864    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l" podStartSLOduration=1.361842964 podStartE2EDuration="1.361842964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:50.358539102 +0000 UTC m=+858.726128546" watchObservedRunningTime="2026-09-11 14:06:50.361842964 +0000 UTC m=+858.729432410"
Sep 11 14:06:51.245480 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:51.245450    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:06:54.328478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.328439    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"]
Sep 11 14:06:54.331769 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.331754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.334371 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.334347    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-80fb45\""
Sep 11 14:06:54.334467 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.334454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-80fb45-dockercfg-rrgtr\""
Sep 11 14:06:54.341584 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.341558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"]
Sep 11 14:06:54.392005 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.391964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745de085-2ec9-46c9-9f37-507dbd5e299e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.392179 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.392083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6xg\" (UniqueName: \"kubernetes.io/projected/745de085-2ec9-46c9-9f37-507dbd5e299e-kube-api-access-lb6xg\") pod \"payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.493251 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.493213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745de085-2ec9-46c9-9f37-507dbd5e299e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.493428 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.493271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6xg\" (UniqueName: \"kubernetes.io/projected/745de085-2ec9-46c9-9f37-507dbd5e299e-kube-api-access-lb6xg\") pod \"payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.493943 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.493896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745de085-2ec9-46c9-9f37-507dbd5e299e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.502040 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.502015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6xg\" (UniqueName: \"kubernetes.io/projected/745de085-2ec9-46c9-9f37-507dbd5e299e-kube-api-access-lb6xg\") pod \"payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.642826 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.642738    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:54.913968 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:54.913944    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"]
Sep 11 14:06:54.916629 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:06:54.916601    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745de085_2ec9_46c9_9f37_507dbd5e299e.slice/crio-072b80dc7e2cedf01b9ed511008e6ccf8005485eec619c1305a307cd21c801af WatchSource:0}: Error finding container 072b80dc7e2cedf01b9ed511008e6ccf8005485eec619c1305a307cd21c801af: Status 404 returned error can't find the container with id 072b80dc7e2cedf01b9ed511008e6ccf8005485eec619c1305a307cd21c801af
Sep 11 14:06:55.316399 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:55.316362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn" event={"ID":"745de085-2ec9-46c9-9f37-507dbd5e299e","Type":"ContainerStarted","Data":"727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88"}
Sep 11 14:06:55.316580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:55.316423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn" event={"ID":"745de085-2ec9-46c9-9f37-507dbd5e299e","Type":"ContainerStarted","Data":"072b80dc7e2cedf01b9ed511008e6ccf8005485eec619c1305a307cd21c801af"}
Sep 11 14:06:55.316580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:55.316449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:06:55.335322 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:55.335275    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn" podStartSLOduration=1.335259615 podStartE2EDuration="1.335259615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:55.334404266 +0000 UTC m=+863.701993708" watchObservedRunningTime="2026-09-11 14:06:55.335259615 +0000 UTC m=+863.702849058"
Sep 11 14:06:59.288899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:59.288866    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:06:59.876100 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:59.876059    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:59.876310 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:59.876133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:06:59.888291 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:06:59.888268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:07:00.349722 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:00.349695    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:07:06.323088 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:06.323055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:07:06.799061 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:06.798977    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:07:17.411813 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:17.411779    2561 generic.go:356] "Generic (PLEG): container finished" podID="872f71a3-fac8-463b-928e-76d5e22c0636" containerID="22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805" exitCode=0
Sep 11 14:07:17.412272 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:17.411831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" event={"ID":"872f71a3-fac8-463b-928e-76d5e22c0636","Type":"ContainerDied","Data":"22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"}
Sep 11 14:07:17.412272 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:17.412203    2561 scope.go:117] "RemoveContainer" containerID="22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"
Sep 11 14:07:17.413183 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:17.413154    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:07:17.413247 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:17.413219    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:07:17.413283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:17.413249    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:07:18.423164 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:18.423123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" event={"ID":"872f71a3-fac8-463b-928e-76d5e22c0636","Type":"ContainerStarted","Data":"159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7"}
Sep 11 14:07:19.272454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:19.272408    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:07:19.273714 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:19.273685    2561 patch_prober.go:28] interesting pod/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:07:19.273849 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:19.273782    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:07:20.272261 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:20.272224    2561 patch_prober.go:28] interesting pod/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:07:20.272667 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:20.272314    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:07:20.293787 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:20.293762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:07:21.272103 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.272070    2561 patch_prober.go:28] interesting pod/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:07:21.272286 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.272149    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:07:21.381516 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.381485    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"]
Sep 11 14:07:21.386740 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.386723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:21.390884 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.390865    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:07:21.391010 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.390989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:07:21.396865 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.396842    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"]
Sep 11 14:07:21.448135 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.448087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41056a0a-886b-4aa3-8b71-e429b054186c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw\" (UID: \"41056a0a-886b-4aa3-8b71-e429b054186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:21.548819 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.548722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41056a0a-886b-4aa3-8b71-e429b054186c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw\" (UID: \"41056a0a-886b-4aa3-8b71-e429b054186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:21.549656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.549626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41056a0a-886b-4aa3-8b71-e429b054186c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw\" (UID: \"41056a0a-886b-4aa3-8b71-e429b054186c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:21.716893 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:21.716855    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:22.009325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:22.009300    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"]
Sep 11 14:07:22.011687 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:07:22.011656    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41056a0a_886b_4aa3_8b71_e429b054186c.slice/crio-85251f8e9594b17dcd1395edc2100f80345e56c2ce082bbbc027163660b421e7 WatchSource:0}: Error finding container 85251f8e9594b17dcd1395edc2100f80345e56c2ce082bbbc027163660b421e7: Status 404 returned error can't find the container with id 85251f8e9594b17dcd1395edc2100f80345e56c2ce082bbbc027163660b421e7
Sep 11 14:07:22.272551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:22.272464    2561 patch_prober.go:28] interesting pod/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:07:22.272551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:22.272531    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:07:22.452878 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:22.452843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw" event={"ID":"41056a0a-886b-4aa3-8b71-e429b054186c","Type":"ContainerStarted","Data":"85251f8e9594b17dcd1395edc2100f80345e56c2ce082bbbc027163660b421e7"}
Sep 11 14:07:23.272922 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:23.272885    2561 patch_prober.go:28] interesting pod/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:07:23.273304 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:23.272959    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:07:24.276318 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:24.276235    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:07:24.277267 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:24.277249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:07:29.485146 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.485062    2561 generic.go:356] "Generic (PLEG): container finished" podID="41056a0a-886b-4aa3-8b71-e429b054186c" containerID="fab6bd2fe7be4d3cf62ae139dbb7204b08b4191ebdb87ab228da008916ebfbd1" exitCode=0
Sep 11 14:07:29.485518 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.485148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw" event={"ID":"41056a0a-886b-4aa3-8b71-e429b054186c","Type":"ContainerDied","Data":"fab6bd2fe7be4d3cf62ae139dbb7204b08b4191ebdb87ab228da008916ebfbd1"}
Sep 11 14:07:29.780042 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.779962    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"]
Sep 11 14:07:29.783256 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.783237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.785981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.785959    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-918362dd-kserve-self-signed-certs\""
Sep 11 14:07:29.797255 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.797228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"]
Sep 11 14:07:29.820188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.820161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1486892-5bb1-4d19-8063-14005481cef7-tls-certs\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.820315 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.820192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-model-cache\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.820315 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.820265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-tmp-dir\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.820315 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.820296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8vm\" (UniqueName: \"kubernetes.io/projected/f1486892-5bb1-4d19-8063-14005481cef7-kube-api-access-lq8vm\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.820420 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.820347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-home\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.820420 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.820378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-dshm\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.921791 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.921752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-tmp-dir\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.921791 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.921791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8vm\" (UniqueName: \"kubernetes.io/projected/f1486892-5bb1-4d19-8063-14005481cef7-kube-api-access-lq8vm\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922033 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.921816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-home\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922033 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.921831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-dshm\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922033 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.922011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1486892-5bb1-4d19-8063-14005481cef7-tls-certs\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922180 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.922053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-model-cache\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922286 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.922265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-home\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922344 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.922265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-tmp-dir\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.922433 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.922413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-model-cache\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.924320 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.924300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-dshm\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.924421 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.924377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1486892-5bb1-4d19-8063-14005481cef7-tls-certs\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:29.929995 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:29.929977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8vm\" (UniqueName: \"kubernetes.io/projected/f1486892-5bb1-4d19-8063-14005481cef7-kube-api-access-lq8vm\") pod \"e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") " pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:30.092657 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.092625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:30.362500 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.362372    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"]
Sep 11 14:07:30.365108 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:07:30.365074    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1486892_5bb1_4d19_8063_14005481cef7.slice/crio-079ef36dd16510b7011c1715f084eba0d4e8fb9f4cfe2c08fffa2ec5d67ccb37 WatchSource:0}: Error finding container 079ef36dd16510b7011c1715f084eba0d4e8fb9f4cfe2c08fffa2ec5d67ccb37: Status 404 returned error can't find the container with id 079ef36dd16510b7011c1715f084eba0d4e8fb9f4cfe2c08fffa2ec5d67ccb37
Sep 11 14:07:30.492110 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.492072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2" event={"ID":"f1486892-5bb1-4d19-8063-14005481cef7","Type":"ContainerStarted","Data":"1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224"}
Sep 11 14:07:30.492512 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.492122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2" event={"ID":"f1486892-5bb1-4d19-8063-14005481cef7","Type":"ContainerStarted","Data":"079ef36dd16510b7011c1715f084eba0d4e8fb9f4cfe2c08fffa2ec5d67ccb37"}
Sep 11 14:07:30.511159 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.511104    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2" podStartSLOduration=1.5110884009999999 podStartE2EDuration="1.511088401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:30.509991588 +0000 UTC m=+898.877581031" watchObservedRunningTime="2026-09-11 14:07:30.511088401 +0000 UTC m=+898.878677844"
Sep 11 14:07:30.607093 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.607066    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:30.627210 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.627126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41056a0a-886b-4aa3-8b71-e429b054186c-maas-api-service-ca\") pod \"41056a0a-886b-4aa3-8b71-e429b054186c\" (UID: \"41056a0a-886b-4aa3-8b71-e429b054186c\") "
Sep 11 14:07:30.627483 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.627459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41056a0a-886b-4aa3-8b71-e429b054186c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41056a0a-886b-4aa3-8b71-e429b054186c" (UID: "41056a0a-886b-4aa3-8b71-e429b054186c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:07:30.728038 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:30.728002    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41056a0a-886b-4aa3-8b71-e429b054186c-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:31.498217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:31.498168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw" event={"ID":"41056a0a-886b-4aa3-8b71-e429b054186c","Type":"ContainerDied","Data":"85251f8e9594b17dcd1395edc2100f80345e56c2ce082bbbc027163660b421e7"}
Sep 11 14:07:31.498595 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:31.498235    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85251f8e9594b17dcd1395edc2100f80345e56c2ce082bbbc027163660b421e7"
Sep 11 14:07:31.498595 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:31.498192    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"
Sep 11 14:07:32.118319 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:32.118294    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:07:32.118697 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:32.118680    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:07:33.421763 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:33.421717    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"]
Sep 11 14:07:33.431140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:33.431112    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e6230ad-7zpgw"]
Sep 11 14:07:33.577580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:33.577551    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"]
Sep 11 14:07:33.577824 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:33.577800    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy" containerID="cri-o://159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7" gracePeriod=30
Sep 11 14:07:34.116491 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:34.116452    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41056a0a-886b-4aa3-8b71-e429b054186c" path="/var/lib/kubelet/pods/41056a0a-886b-4aa3-8b71-e429b054186c/volumes"
Sep 11 14:07:35.912122 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:35.912084    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:07:38.823747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.823721    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:07:38.900076 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.899991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872f71a3-fac8-463b-928e-76d5e22c0636-istio-podinfo\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900076 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-data\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900076 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kc28\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-kube-api-access-6kc28\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900076 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900079    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-envoy\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-socket\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900146    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872f71a3-fac8-463b-928e-76d5e22c0636-istiod-ca-cert\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-istio-token\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-credential-socket\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-certs\") pod \"872f71a3-fac8-463b-928e-76d5e22c0636\" (UID: \"872f71a3-fac8-463b-928e-76d5e22c0636\") "
Sep 11 14:07:38.900392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-data" (OuterVolumeSpecName: "istio-data") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:07:38.900999 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.900963    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-data\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:38.901143 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.901057    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872f71a3-fac8-463b-928e-76d5e22c0636-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:07:38.902445 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.901342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:07:38.902445 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.901384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:07:38.902445 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.901562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:07:38.909136 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.909095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872f71a3-fac8-463b-928e-76d5e22c0636-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:07:38.909288 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.909202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-istio-token" (OuterVolumeSpecName: "istio-token") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:07:38.909288 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.909247    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-kube-api-access-6kc28" (OuterVolumeSpecName: "kube-api-access-6kc28") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "kube-api-access-6kc28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:07:38.909288 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:38.909261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "872f71a3-fac8-463b-928e-76d5e22c0636" (UID: "872f71a3-fac8-463b-928e-76d5e22c0636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:07:39.002159 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002122    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872f71a3-fac8-463b-928e-76d5e22c0636-istiod-ca-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002159 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002156    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-istio-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002159 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002167    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-credential-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002180    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002192    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872f71a3-fac8-463b-928e-76d5e22c0636-istio-podinfo\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002203    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kc28\" (UniqueName: \"kubernetes.io/projected/872f71a3-fac8-463b-928e-76d5e22c0636-kube-api-access-6kc28\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002214    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-istio-envoy\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.002395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.002225    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872f71a3-fac8-463b-928e-76d5e22c0636-workload-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:07:39.536529 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.536485    2561 generic.go:356] "Generic (PLEG): container finished" podID="872f71a3-fac8-463b-928e-76d5e22c0636" containerID="159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7" exitCode=0
Sep 11 14:07:39.536724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.536539    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"
Sep 11 14:07:39.536724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.536570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" event={"ID":"872f71a3-fac8-463b-928e-76d5e22c0636","Type":"ContainerDied","Data":"159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7"}
Sep 11 14:07:39.536724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.536639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq" event={"ID":"872f71a3-fac8-463b-928e-76d5e22c0636","Type":"ContainerDied","Data":"c4f14168283b33fba86e4d829e2f27596c549b5fa55497debae02302cee2920c"}
Sep 11 14:07:39.536724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.536671    2561 scope.go:117] "RemoveContainer" containerID="159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7"
Sep 11 14:07:39.546053 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.546032    2561 scope.go:117] "RemoveContainer" containerID="22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"
Sep 11 14:07:39.555309 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.555288    2561 scope.go:117] "RemoveContainer" containerID="159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7"
Sep 11 14:07:39.555600 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:07:39.555581    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7\": container with ID starting with 159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7 not found: ID does not exist" containerID="159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7"
Sep 11 14:07:39.555671 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.555608    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7"} err="failed to get container status \"159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7\": rpc error: code = NotFound desc = could not find container \"159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7\": container with ID starting with 159a71f03b161feb5407e2a2eb5e5224031152437cfa307251961afc18e192b7 not found: ID does not exist"
Sep 11 14:07:39.555671 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.555625    2561 scope.go:117] "RemoveContainer" containerID="22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"
Sep 11 14:07:39.555936 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:07:39.555916    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805\": container with ID starting with 22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805 not found: ID does not exist" containerID="22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"
Sep 11 14:07:39.556013 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.555939    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805"} err="failed to get container status \"22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805\": rpc error: code = NotFound desc = could not find container \"22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805\": container with ID starting with 22c19490a3cde04957c66026a12046d6819a7396971a343153d38df7d582c805 not found: ID does not exist"
Sep 11 14:07:39.562827 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.562805    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"]
Sep 11 14:07:39.571236 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.571208    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4e6230ad-openshift-default-77774c79c8-9qtfq"]
Sep 11 14:07:39.653613 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653576    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"]
Sep 11 14:07:39.653882 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653867    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41056a0a-886b-4aa3-8b71-e429b054186c" containerName="revoke-keys"
Sep 11 14:07:39.653882 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653882    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41056a0a-886b-4aa3-8b71-e429b054186c" containerName="revoke-keys"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653901    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653918    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653935    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653943    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.653994    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.654004    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" containerName="istio-proxy"
Sep 11 14:07:39.654032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.654012    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41056a0a-886b-4aa3-8b71-e429b054186c" containerName="revoke-keys"
Sep 11 14:07:39.658433 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.658412    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.666027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.666004    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-42670434-dockercfg-gsswz\""
Sep 11 14:07:39.666284 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.666267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-42670434\""
Sep 11 14:07:39.677772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.677747    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"]
Sep 11 14:07:39.708218 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.708183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1eef52-8250-4f21-b0e7-f5ba82aba62e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.708362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.708262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmvg\" (UniqueName: \"kubernetes.io/projected/af1eef52-8250-4f21-b0e7-f5ba82aba62e-kube-api-access-dtmvg\") pod \"payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.715262 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.715235    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"]
Sep 11 14:07:39.718351 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.718336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.739139 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.739113    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"]
Sep 11 14:07:39.809666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.809638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1eef52-8250-4f21-b0e7-f5ba82aba62e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.809818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.809686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmt9\" (UniqueName: \"kubernetes.io/projected/4dc66d3c-1b44-4e16-9734-ce621a589c93-kube-api-access-4hmt9\") pod \"payload-processing-e2e-migrate-42670434-856bd87574-pdzds\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") " pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.809818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.809716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmvg\" (UniqueName: \"kubernetes.io/projected/af1eef52-8250-4f21-b0e7-f5ba82aba62e-kube-api-access-dtmvg\") pod \"payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.809818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.809747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc66d3c-1b44-4e16-9734-ce621a589c93-plugins-config-volume\") pod \"payload-processing-e2e-migrate-42670434-856bd87574-pdzds\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") " pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.810400 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.810382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1eef52-8250-4f21-b0e7-f5ba82aba62e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.824857 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.824822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmvg\" (UniqueName: \"kubernetes.io/projected/af1eef52-8250-4f21-b0e7-f5ba82aba62e-kube-api-access-dtmvg\") pod \"payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:39.910892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.910839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc66d3c-1b44-4e16-9734-ce621a589c93-plugins-config-volume\") pod \"payload-processing-e2e-migrate-42670434-856bd87574-pdzds\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") " pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.911080 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.910975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmt9\" (UniqueName: \"kubernetes.io/projected/4dc66d3c-1b44-4e16-9734-ce621a589c93-kube-api-access-4hmt9\") pod \"payload-processing-e2e-migrate-42670434-856bd87574-pdzds\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") " pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.911607 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.911587    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc66d3c-1b44-4e16-9734-ce621a589c93-plugins-config-volume\") pod \"payload-processing-e2e-migrate-42670434-856bd87574-pdzds\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") " pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.928582 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.928555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmt9\" (UniqueName: \"kubernetes.io/projected/4dc66d3c-1b44-4e16-9734-ce621a589c93-kube-api-access-4hmt9\") pod \"payload-processing-e2e-migrate-42670434-856bd87574-pdzds\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") " pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:39.971438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:39.971399    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:40.029125 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.029065    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:40.092986 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.092957    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:40.093557 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.093512    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:40.117490 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.117460    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872f71a3-fac8-463b-928e-76d5e22c0636" path="/var/lib/kubelet/pods/872f71a3-fac8-463b-928e-76d5e22c0636/volumes"
Sep 11 14:07:40.117780 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.117767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:40.248611 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.248586    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"]
Sep 11 14:07:40.251125 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:07:40.251103    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1eef52_8250_4f21_b0e7_f5ba82aba62e.slice/crio-c2cedd9d83d1dfbcab328b77547695393cb4f343160472c1216fd6a021116a90 WatchSource:0}: Error finding container c2cedd9d83d1dfbcab328b77547695393cb4f343160472c1216fd6a021116a90: Status 404 returned error can't find the container with id c2cedd9d83d1dfbcab328b77547695393cb4f343160472c1216fd6a021116a90
Sep 11 14:07:40.319702 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.319676    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"]
Sep 11 14:07:40.321382 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:07:40.321359    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc66d3c_1b44_4e16_9734_ce621a589c93.slice/crio-914ba3e7e085990c944c88abc54b73d70493ae08d02ce5f2f13356c778fe42c4 WatchSource:0}: Error finding container 914ba3e7e085990c944c88abc54b73d70493ae08d02ce5f2f13356c778fe42c4: Status 404 returned error can't find the container with id 914ba3e7e085990c944c88abc54b73d70493ae08d02ce5f2f13356c778fe42c4
Sep 11 14:07:40.544539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.544504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6" event={"ID":"af1eef52-8250-4f21-b0e7-f5ba82aba62e","Type":"ContainerStarted","Data":"fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e"}
Sep 11 14:07:40.544736 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.544568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6" event={"ID":"af1eef52-8250-4f21-b0e7-f5ba82aba62e","Type":"ContainerStarted","Data":"c2cedd9d83d1dfbcab328b77547695393cb4f343160472c1216fd6a021116a90"}
Sep 11 14:07:40.544736 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.544599    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:40.546186 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.546164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds" event={"ID":"4dc66d3c-1b44-4e16-9734-ce621a589c93","Type":"ContainerStarted","Data":"db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2"}
Sep 11 14:07:40.546300 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.546210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds" event={"ID":"4dc66d3c-1b44-4e16-9734-ce621a589c93","Type":"ContainerStarted","Data":"914ba3e7e085990c944c88abc54b73d70493ae08d02ce5f2f13356c778fe42c4"}
Sep 11 14:07:40.564490 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.564470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:07:40.573221 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.573172    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6" podStartSLOduration=1.573157545 podStartE2EDuration="1.573157545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:40.570955756 +0000 UTC m=+908.938545199" watchObservedRunningTime="2026-09-11 14:07:40.573157545 +0000 UTC m=+908.940747037"
Sep 11 14:07:40.598739 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:40.598639    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds" podStartSLOduration=1.598622957 podStartE2EDuration="1.598622957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:40.596224711 +0000 UTC m=+908.963814148" watchObservedRunningTime="2026-09-11 14:07:40.598622957 +0000 UTC m=+908.966212400"
Sep 11 14:07:41.550890 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:41.550862    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:51.552905 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:51.552875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:07:52.557083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:52.557048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:07:57.540947 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:07:57.540189    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:08:08.370829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.370795    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"]
Sep 11 14:08:08.371300 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.371054    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds" podUID="4dc66d3c-1b44-4e16-9734-ce621a589c93" containerName="payload-processing" containerID="cri-o://db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2" gracePeriod=30
Sep 11 14:08:08.378925 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.378889    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"]
Sep 11 14:08:08.379111 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.379089    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6" podUID="af1eef52-8250-4f21-b0e7-f5ba82aba62e" containerName="payload-pre-processing" containerID="cri-o://fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e" gracePeriod=30
Sep 11 14:08:08.643800 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.643774    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:08:08.646602 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.646580    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:08:08.669516 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.669485    2561 generic.go:356] "Generic (PLEG): container finished" podID="af1eef52-8250-4f21-b0e7-f5ba82aba62e" containerID="fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e" exitCode=0
Sep 11 14:08:08.669516 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.669517    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"
Sep 11 14:08:08.669689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.669576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6" event={"ID":"af1eef52-8250-4f21-b0e7-f5ba82aba62e","Type":"ContainerDied","Data":"fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e"}
Sep 11 14:08:08.669689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.669631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6" event={"ID":"af1eef52-8250-4f21-b0e7-f5ba82aba62e","Type":"ContainerDied","Data":"c2cedd9d83d1dfbcab328b77547695393cb4f343160472c1216fd6a021116a90"}
Sep 11 14:08:08.669689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.669663    2561 scope.go:117] "RemoveContainer" containerID="fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e"
Sep 11 14:08:08.670841 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.670822    2561 generic.go:356] "Generic (PLEG): container finished" podID="4dc66d3c-1b44-4e16-9734-ce621a589c93" containerID="db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2" exitCode=0
Sep 11 14:08:08.670935 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.670847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds" event={"ID":"4dc66d3c-1b44-4e16-9734-ce621a589c93","Type":"ContainerDied","Data":"db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2"}
Sep 11 14:08:08.670935 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.670890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds" event={"ID":"4dc66d3c-1b44-4e16-9734-ce621a589c93","Type":"ContainerDied","Data":"914ba3e7e085990c944c88abc54b73d70493ae08d02ce5f2f13356c778fe42c4"}
Sep 11 14:08:08.670935 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.670858    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"
Sep 11 14:08:08.677506 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.677491    2561 scope.go:117] "RemoveContainer" containerID="fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e"
Sep 11 14:08:08.677821 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:08:08.677789    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e\": container with ID starting with fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e not found: ID does not exist" containerID="fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e"
Sep 11 14:08:08.677903 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.677831    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e"} err="failed to get container status \"fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e\": rpc error: code = NotFound desc = could not find container \"fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e\": container with ID starting with fc705e1854026129c585a6a6e50d802d8eae53f3b7ca56aeef3030a07064d17e not found: ID does not exist"
Sep 11 14:08:08.677903 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.677848    2561 scope.go:117] "RemoveContainer" containerID="db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2"
Sep 11 14:08:08.685210 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.685193    2561 scope.go:117] "RemoveContainer" containerID="db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2"
Sep 11 14:08:08.685478 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:08:08.685461    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2\": container with ID starting with db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2 not found: ID does not exist" containerID="db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2"
Sep 11 14:08:08.685535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.685488    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2"} err="failed to get container status \"db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2\": rpc error: code = NotFound desc = could not find container \"db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2\": container with ID starting with db507cf1f04ef0ebd41fdd12b4ea7b6961235e058a380adc9e3b8be0d47514b2 not found: ID does not exist"
Sep 11 14:08:08.762376 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.762350    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc66d3c-1b44-4e16-9734-ce621a589c93-plugins-config-volume\") pod \"4dc66d3c-1b44-4e16-9734-ce621a589c93\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") "
Sep 11 14:08:08.762530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.762394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hmt9\" (UniqueName: \"kubernetes.io/projected/4dc66d3c-1b44-4e16-9734-ce621a589c93-kube-api-access-4hmt9\") pod \"4dc66d3c-1b44-4e16-9734-ce621a589c93\" (UID: \"4dc66d3c-1b44-4e16-9734-ce621a589c93\") "
Sep 11 14:08:08.762530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.762422    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmvg\" (UniqueName: \"kubernetes.io/projected/af1eef52-8250-4f21-b0e7-f5ba82aba62e-kube-api-access-dtmvg\") pod \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") "
Sep 11 14:08:08.762530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.762498    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1eef52-8250-4f21-b0e7-f5ba82aba62e-plugins-config-volume\") pod \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\" (UID: \"af1eef52-8250-4f21-b0e7-f5ba82aba62e\") "
Sep 11 14:08:08.762898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.762866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1eef52-8250-4f21-b0e7-f5ba82aba62e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af1eef52-8250-4f21-b0e7-f5ba82aba62e" (UID: "af1eef52-8250-4f21-b0e7-f5ba82aba62e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:08:08.763009 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.762896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc66d3c-1b44-4e16-9734-ce621a589c93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dc66d3c-1b44-4e16-9734-ce621a589c93" (UID: "4dc66d3c-1b44-4e16-9734-ce621a589c93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:08:08.764506 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.764485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1eef52-8250-4f21-b0e7-f5ba82aba62e-kube-api-access-dtmvg" (OuterVolumeSpecName: "kube-api-access-dtmvg") pod "af1eef52-8250-4f21-b0e7-f5ba82aba62e" (UID: "af1eef52-8250-4f21-b0e7-f5ba82aba62e"). InnerVolumeSpecName "kube-api-access-dtmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:08:08.764604 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.764527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc66d3c-1b44-4e16-9734-ce621a589c93-kube-api-access-4hmt9" (OuterVolumeSpecName: "kube-api-access-4hmt9") pod "4dc66d3c-1b44-4e16-9734-ce621a589c93" (UID: "4dc66d3c-1b44-4e16-9734-ce621a589c93"). InnerVolumeSpecName "kube-api-access-4hmt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:08:08.863295 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.863264    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1eef52-8250-4f21-b0e7-f5ba82aba62e-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:08.863295 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.863294    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc66d3c-1b44-4e16-9734-ce621a589c93-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:08.863493 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.863310    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmt9\" (UniqueName: \"kubernetes.io/projected/4dc66d3c-1b44-4e16-9734-ce621a589c93-kube-api-access-4hmt9\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:08.863493 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:08.863320    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmvg\" (UniqueName: \"kubernetes.io/projected/af1eef52-8250-4f21-b0e7-f5ba82aba62e-kube-api-access-dtmvg\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:09.008340 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.008311    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"]
Sep 11 14:08:09.015418 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.015396    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-42670434-7cc48bd4fd-28pt6"]
Sep 11 14:08:09.038000 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.037968    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"]
Sep 11 14:08:09.043873 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.043849    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-42670434-856bd87574-pdzds"]
Sep 11 14:08:09.769240 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769204    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"]
Sep 11 14:08:09.769600 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769522    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af1eef52-8250-4f21-b0e7-f5ba82aba62e" containerName="payload-pre-processing"
Sep 11 14:08:09.769600 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769534    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af1eef52-8250-4f21-b0e7-f5ba82aba62e" containerName="payload-pre-processing"
Sep 11 14:08:09.769600 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769544    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc66d3c-1b44-4e16-9734-ce621a589c93" containerName="payload-processing"
Sep 11 14:08:09.769600 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769550    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc66d3c-1b44-4e16-9734-ce621a589c93" containerName="payload-processing"
Sep 11 14:08:09.769724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769607    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af1eef52-8250-4f21-b0e7-f5ba82aba62e" containerName="payload-pre-processing"
Sep 11 14:08:09.769724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.769615    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc66d3c-1b44-4e16-9734-ce621a589c93" containerName="payload-processing"
Sep 11 14:08:09.773975 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.773956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.778383 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.778363    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:08:09.778772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.778756    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-383709f1-openshift-default-dockercfg-mbfc5\""
Sep 11 14:08:09.800871 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.800851    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"]
Sep 11 14:08:09.873048 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-socket\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873048 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-credential-socket\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873074    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgp4l\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-kube-api-access-zgp4l\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-token\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istiod-ca-cert\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-certs\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-envoy\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873446 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-podinfo\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.873446 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.873326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-data\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-data\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-socket\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-credential-socket\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgp4l\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-kube-api-access-zgp4l\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-token\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istiod-ca-cert\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-certs\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-envoy\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.974892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.974820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-podinfo\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.975236 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.975027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-credential-socket\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.975236 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.975128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-socket\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.975236 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.975194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-data\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.975331 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.975261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-certs\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.975611 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.975592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istiod-ca-cert\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.977202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.977182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-envoy\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:09.977256 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:09.977192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-podinfo\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:10.017660 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.017597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgp4l\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-kube-api-access-zgp4l\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:10.037818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.037794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-token\") pod \"e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") " pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:10.083420 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.083395    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:10.116665 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.116625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc66d3c-1b44-4e16-9734-ce621a589c93" path="/var/lib/kubelet/pods/4dc66d3c-1b44-4e16-9734-ce621a589c93/volumes"
Sep 11 14:08:10.117089 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.117069    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af1eef52-8250-4f21-b0e7-f5ba82aba62e" path="/var/lib/kubelet/pods/af1eef52-8250-4f21-b0e7-f5ba82aba62e/volumes"
Sep 11 14:08:10.384920 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.384880    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"]
Sep 11 14:08:10.386404 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:08:10.386373    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c7cebc_1d51_4ce8_a8ee_fb353ff8168d.slice/crio-9b82b448562d8b21aeb7f1ee8c91982bf3708d8bb24e0a6b6497672de3ef531d WatchSource:0}: Error finding container 9b82b448562d8b21aeb7f1ee8c91982bf3708d8bb24e0a6b6497672de3ef531d: Status 404 returned error can't find the container with id 9b82b448562d8b21aeb7f1ee8c91982bf3708d8bb24e0a6b6497672de3ef531d
Sep 11 14:08:10.388644 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.388609    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:08:10.388704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.388692    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:08:10.388746 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.388736    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:08:10.684076 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.683975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" event={"ID":"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d","Type":"ContainerStarted","Data":"de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"}
Sep 11 14:08:10.684076 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.684042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" event={"ID":"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d","Type":"ContainerStarted","Data":"9b82b448562d8b21aeb7f1ee8c91982bf3708d8bb24e0a6b6497672de3ef531d"}
Sep 11 14:08:10.709556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:10.709479    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podStartSLOduration=1.709456637 podStartE2EDuration="1.709456637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:10.70785405 +0000 UTC m=+939.075443492" watchObservedRunningTime="2026-09-11 14:08:10.709456637 +0000 UTC m=+939.077046081"
Sep 11 14:08:11.084230 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:11.084199    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:11.089725 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:11.089703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:11.688630 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:11.688591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:11.689507 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:11.689486    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:14.806016 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:14.805986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:08:15.312257 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.312215    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"]
Sep 11 14:08:15.315761 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.315738    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.319184 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.319158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-383709f1\""
Sep 11 14:08:15.320350 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.320329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-383709f1-dockercfg-dzkqg\""
Sep 11 14:08:15.322722 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.322694    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"]
Sep 11 14:08:15.326158 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.326131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.336027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.336007    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"]
Sep 11 14:08:15.344095 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.344071    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"]
Sep 11 14:08:15.424588 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.424552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8529d3d2-00cf-4697-9225-eb40f7afb30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.424766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.424641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpc6\" (UniqueName: \"kubernetes.io/projected/8529d3d2-00cf-4697-9225-eb40f7afb30e-kube-api-access-5cpc6\") pod \"payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.525520 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.525486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8529d3d2-00cf-4697-9225-eb40f7afb30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.525678 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.525646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllzb\" (UniqueName: \"kubernetes.io/projected/817ee4af-2e6a-4bda-b46f-7aca14b41c62-kube-api-access-tllzb\") pod \"payload-processing-e2e-ait-383709f1-58d76669d5-mxtff\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") " pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.525678 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.525666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817ee4af-2e6a-4bda-b46f-7aca14b41c62-plugins-config-volume\") pod \"payload-processing-e2e-ait-383709f1-58d76669d5-mxtff\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") " pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.525759 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.525693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpc6\" (UniqueName: \"kubernetes.io/projected/8529d3d2-00cf-4697-9225-eb40f7afb30e-kube-api-access-5cpc6\") pod \"payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.526270 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.526251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8529d3d2-00cf-4697-9225-eb40f7afb30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.535349 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.535328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpc6\" (UniqueName: \"kubernetes.io/projected/8529d3d2-00cf-4697-9225-eb40f7afb30e-kube-api-access-5cpc6\") pod \"payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.626264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.626182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllzb\" (UniqueName: \"kubernetes.io/projected/817ee4af-2e6a-4bda-b46f-7aca14b41c62-kube-api-access-tllzb\") pod \"payload-processing-e2e-ait-383709f1-58d76669d5-mxtff\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") " pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.626264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.626219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817ee4af-2e6a-4bda-b46f-7aca14b41c62-plugins-config-volume\") pod \"payload-processing-e2e-ait-383709f1-58d76669d5-mxtff\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") " pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.626872 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.626853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817ee4af-2e6a-4bda-b46f-7aca14b41c62-plugins-config-volume\") pod \"payload-processing-e2e-ait-383709f1-58d76669d5-mxtff\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") " pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.626990 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.626977    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:15.635804 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.635782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllzb\" (UniqueName: \"kubernetes.io/projected/817ee4af-2e6a-4bda-b46f-7aca14b41c62-kube-api-access-tllzb\") pod \"payload-processing-e2e-ait-383709f1-58d76669d5-mxtff\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") " pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:15.903811 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.903786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"]
Sep 11 14:08:15.906045 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:08:15.906015    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8529d3d2_00cf_4697_9225_eb40f7afb30e.slice/crio-f76435a59980c58a104b696e04275d92c61223266e166185ca3f46ae203120e2 WatchSource:0}: Error finding container f76435a59980c58a104b696e04275d92c61223266e166185ca3f46ae203120e2: Status 404 returned error can't find the container with id f76435a59980c58a104b696e04275d92c61223266e166185ca3f46ae203120e2
Sep 11 14:08:15.935152 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:15.935125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:16.230392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.230366    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"]
Sep 11 14:08:16.232458 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:08:16.232422    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817ee4af_2e6a_4bda_b46f_7aca14b41c62.slice/crio-bbac66d1ef360871c6140b5217c9b7b0d5c8d41b5110a630d8e251ff71e8c35e WatchSource:0}: Error finding container bbac66d1ef360871c6140b5217c9b7b0d5c8d41b5110a630d8e251ff71e8c35e: Status 404 returned error can't find the container with id bbac66d1ef360871c6140b5217c9b7b0d5c8d41b5110a630d8e251ff71e8c35e
Sep 11 14:08:16.710658 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.710623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff" event={"ID":"817ee4af-2e6a-4bda-b46f-7aca14b41c62","Type":"ContainerStarted","Data":"93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686"}
Sep 11 14:08:16.710871 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.710682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff" event={"ID":"817ee4af-2e6a-4bda-b46f-7aca14b41c62","Type":"ContainerStarted","Data":"bbac66d1ef360871c6140b5217c9b7b0d5c8d41b5110a630d8e251ff71e8c35e"}
Sep 11 14:08:16.710871 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.710711    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:16.712197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.712170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf" event={"ID":"8529d3d2-00cf-4697-9225-eb40f7afb30e","Type":"ContainerStarted","Data":"2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328"}
Sep 11 14:08:16.712294 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.712220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf" event={"ID":"8529d3d2-00cf-4697-9225-eb40f7afb30e","Type":"ContainerStarted","Data":"f76435a59980c58a104b696e04275d92c61223266e166185ca3f46ae203120e2"}
Sep 11 14:08:16.712350 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.712308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:16.735738 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:16.735689    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff" podStartSLOduration=1.735670932 podStartE2EDuration="1.735670932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:16.735443464 +0000 UTC m=+945.103032906" watchObservedRunningTime="2026-09-11 14:08:16.735670932 +0000 UTC m=+945.103260390"
Sep 11 14:08:27.719035 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:27.719003    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:27.719531 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:27.719177    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:27.747896 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:27.747834    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf" podStartSLOduration=12.747812843 podStartE2EDuration="12.747812843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:16.76776494 +0000 UTC m=+945.135354382" watchObservedRunningTime="2026-09-11 14:08:27.747812843 +0000 UTC m=+956.115402287"
Sep 11 14:08:33.620260 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.620224    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:08:33.790285 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.790207    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerID="de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b" exitCode=0
Sep 11 14:08:33.790285 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.790255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" event={"ID":"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d","Type":"ContainerDied","Data":"de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"}
Sep 11 14:08:33.790646 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.790628    2561 scope.go:117] "RemoveContainer" containerID="de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"
Sep 11 14:08:33.791718 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.791691    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:08:33.791779 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.791766    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:08:33.791817 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:33.791805    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:08:34.796621 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:34.796584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" event={"ID":"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d","Type":"ContainerStarted","Data":"19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21"}
Sep 11 14:08:35.084138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:35.084102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:35.084368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:35.084341    2561 patch_prober.go:28] interesting pod/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 14:08:35.084448 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:35.084401    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 14:08:36.084618 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:36.084574    2561 patch_prober.go:28] interesting pod/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 14:08:36.085045 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:36.084673    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 14:08:37.083986 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:37.083950    2561 patch_prober.go:28] interesting pod/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 14:08:37.084163 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:37.084006    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 14:08:38.083886 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:38.083842    2561 patch_prober.go:28] interesting pod/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 14:08:38.084330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:38.083938    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 14:08:39.084820 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:39.084779    2561 patch_prober.go:28] interesting pod/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 14:08:39.085243 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:39.084844    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 14:08:40.158428 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:40.158397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:40.158966 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:40.158877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:40.160210 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:40.160192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:42.727408 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.727372    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"]
Sep 11 14:08:42.730699 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.730682    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.738335 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.738316    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-abfca2\"/\"kube-root-ca.crt\""
Sep 11 14:08:42.738427 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.738373    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-abfca2\"/\"test-model-abfca2-kserve-self-signed-certs\""
Sep 11 14:08:42.738427 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.738390    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-abfca2\"/\"default-dockercfg-s4646\""
Sep 11 14:08:42.738930 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.738902    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-abfca2\"/\"openshift-service-ca.crt\""
Sep 11 14:08:42.761347 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.761313    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"]
Sep 11 14:08:42.886555 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.886522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-home\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.886721 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.886565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-model-cache\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.886721 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.886602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-dshm\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.886721 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.886670    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6wtz\" (UniqueName: \"kubernetes.io/projected/5b52c67b-2d36-4060-a768-f483ad246ab2-kube-api-access-m6wtz\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.886829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.886723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b52c67b-2d36-4060-a768-f483ad246ab2-tls-certs\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.886829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.886752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-tmp-dir\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6wtz\" (UniqueName: \"kubernetes.io/projected/5b52c67b-2d36-4060-a768-f483ad246ab2-kube-api-access-m6wtz\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b52c67b-2d36-4060-a768-f483ad246ab2-tls-certs\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-tmp-dir\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-home\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-model-cache\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-dshm\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988519 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-tmp-dir\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988579 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-home\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.988658 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.988636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-model-cache\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.990359 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.990328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-dshm\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:42.990626 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:42.990610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b52c67b-2d36-4060-a768-f483ad246ab2-tls-certs\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:43.009617 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:43.009595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6wtz\" (UniqueName: \"kubernetes.io/projected/5b52c67b-2d36-4060-a768-f483ad246ab2-kube-api-access-m6wtz\") pod \"test-model-abfca2-kserve-75c54d8d55-wgrb2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") " pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:43.039739 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:43.039710    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:43.319457 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:43.319434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"]
Sep 11 14:08:43.321340 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:08:43.321319    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b52c67b_2d36_4060_a768_f483ad246ab2.slice/crio-619e7b31551ad24877b66a3c906681d729d06e0fe17d76038b9b28afb7cb40ce WatchSource:0}: Error finding container 619e7b31551ad24877b66a3c906681d729d06e0fe17d76038b9b28afb7cb40ce: Status 404 returned error can't find the container with id 619e7b31551ad24877b66a3c906681d729d06e0fe17d76038b9b28afb7cb40ce
Sep 11 14:08:43.836811 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:43.836775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" event={"ID":"5b52c67b-2d36-4060-a768-f483ad246ab2","Type":"ContainerStarted","Data":"cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1"}
Sep 11 14:08:43.837219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:43.836824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" event={"ID":"5b52c67b-2d36-4060-a768-f483ad246ab2","Type":"ContainerStarted","Data":"619e7b31551ad24877b66a3c906681d729d06e0fe17d76038b9b28afb7cb40ce"}
Sep 11 14:08:43.933579 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:43.933527    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" podStartSLOduration=1.9335113659999998 podStartE2EDuration="1.933511366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:43.929531247 +0000 UTC m=+972.297120713" watchObservedRunningTime="2026-09-11 14:08:43.933511366 +0000 UTC m=+972.301100843"
Sep 11 14:08:44.048734 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.048700    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"]
Sep 11 14:08:44.048967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.048928    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff" podUID="817ee4af-2e6a-4bda-b46f-7aca14b41c62" containerName="payload-processing" containerID="cri-o://93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686" gracePeriod=30
Sep 11 14:08:44.068472 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.068448    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"]
Sep 11 14:08:44.068836 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.068808    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf" podUID="8529d3d2-00cf-4697-9225-eb40f7afb30e" containerName="payload-pre-processing" containerID="cri-o://2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328" gracePeriod=30
Sep 11 14:08:44.315743 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.315714    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:44.338335 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.338312    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:44.399944 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.399850    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817ee4af-2e6a-4bda-b46f-7aca14b41c62-plugins-config-volume\") pod \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") "
Sep 11 14:08:44.399944 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.399888    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tllzb\" (UniqueName: \"kubernetes.io/projected/817ee4af-2e6a-4bda-b46f-7aca14b41c62-kube-api-access-tllzb\") pod \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\" (UID: \"817ee4af-2e6a-4bda-b46f-7aca14b41c62\") "
Sep 11 14:08:44.400235 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.400212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817ee4af-2e6a-4bda-b46f-7aca14b41c62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "817ee4af-2e6a-4bda-b46f-7aca14b41c62" (UID: "817ee4af-2e6a-4bda-b46f-7aca14b41c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:08:44.401982 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.401955    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817ee4af-2e6a-4bda-b46f-7aca14b41c62-kube-api-access-tllzb" (OuterVolumeSpecName: "kube-api-access-tllzb") pod "817ee4af-2e6a-4bda-b46f-7aca14b41c62" (UID: "817ee4af-2e6a-4bda-b46f-7aca14b41c62"). InnerVolumeSpecName "kube-api-access-tllzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:08:44.500797 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.500769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8529d3d2-00cf-4697-9225-eb40f7afb30e-plugins-config-volume\") pod \"8529d3d2-00cf-4697-9225-eb40f7afb30e\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") "
Sep 11 14:08:44.501004 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.500863    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cpc6\" (UniqueName: \"kubernetes.io/projected/8529d3d2-00cf-4697-9225-eb40f7afb30e-kube-api-access-5cpc6\") pod \"8529d3d2-00cf-4697-9225-eb40f7afb30e\" (UID: \"8529d3d2-00cf-4697-9225-eb40f7afb30e\") "
Sep 11 14:08:44.501090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.501029    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817ee4af-2e6a-4bda-b46f-7aca14b41c62-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:44.501090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.501040    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tllzb\" (UniqueName: \"kubernetes.io/projected/817ee4af-2e6a-4bda-b46f-7aca14b41c62-kube-api-access-tllzb\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:44.501212 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.501157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8529d3d2-00cf-4697-9225-eb40f7afb30e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8529d3d2-00cf-4697-9225-eb40f7afb30e" (UID: "8529d3d2-00cf-4697-9225-eb40f7afb30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:08:44.502898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.502875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8529d3d2-00cf-4697-9225-eb40f7afb30e-kube-api-access-5cpc6" (OuterVolumeSpecName: "kube-api-access-5cpc6") pod "8529d3d2-00cf-4697-9225-eb40f7afb30e" (UID: "8529d3d2-00cf-4697-9225-eb40f7afb30e"). InnerVolumeSpecName "kube-api-access-5cpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:08:44.601680 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.601650    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpc6\" (UniqueName: \"kubernetes.io/projected/8529d3d2-00cf-4697-9225-eb40f7afb30e-kube-api-access-5cpc6\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:44.601680 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.601676    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8529d3d2-00cf-4697-9225-eb40f7afb30e-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:44.842266 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.842231    2561 generic.go:356] "Generic (PLEG): container finished" podID="8529d3d2-00cf-4697-9225-eb40f7afb30e" containerID="2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328" exitCode=0
Sep 11 14:08:44.842740 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.842278    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"
Sep 11 14:08:44.842740 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.842318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf" event={"ID":"8529d3d2-00cf-4697-9225-eb40f7afb30e","Type":"ContainerDied","Data":"2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328"}
Sep 11 14:08:44.842740 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.842369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf" event={"ID":"8529d3d2-00cf-4697-9225-eb40f7afb30e","Type":"ContainerDied","Data":"f76435a59980c58a104b696e04275d92c61223266e166185ca3f46ae203120e2"}
Sep 11 14:08:44.842740 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.842393    2561 scope.go:117] "RemoveContainer" containerID="2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328"
Sep 11 14:08:44.843724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.843703    2561 generic.go:356] "Generic (PLEG): container finished" podID="817ee4af-2e6a-4bda-b46f-7aca14b41c62" containerID="93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686" exitCode=0
Sep 11 14:08:44.843814 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.843746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"
Sep 11 14:08:44.843859 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.843805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff" event={"ID":"817ee4af-2e6a-4bda-b46f-7aca14b41c62","Type":"ContainerDied","Data":"93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686"}
Sep 11 14:08:44.843859 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.843852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff" event={"ID":"817ee4af-2e6a-4bda-b46f-7aca14b41c62","Type":"ContainerDied","Data":"bbac66d1ef360871c6140b5217c9b7b0d5c8d41b5110a630d8e251ff71e8c35e"}
Sep 11 14:08:44.852484 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.852470    2561 scope.go:117] "RemoveContainer" containerID="2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328"
Sep 11 14:08:44.852764 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:08:44.852747    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328\": container with ID starting with 2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328 not found: ID does not exist" containerID="2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328"
Sep 11 14:08:44.852822 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.852771    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328"} err="failed to get container status \"2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328\": rpc error: code = NotFound desc = could not find container \"2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328\": container with ID starting with 2c59a0e16750dd87d4b0edfd5c2530bc9b6bbaca357bf4771c3e4faac878c328 not found: ID does not exist"
Sep 11 14:08:44.852822 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.852786    2561 scope.go:117] "RemoveContainer" containerID="93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686"
Sep 11 14:08:44.859760 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.859745    2561 scope.go:117] "RemoveContainer" containerID="93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686"
Sep 11 14:08:44.860021 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:08:44.860003    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686\": container with ID starting with 93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686 not found: ID does not exist" containerID="93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686"
Sep 11 14:08:44.860083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.860025    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686"} err="failed to get container status \"93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686\": rpc error: code = NotFound desc = could not find container \"93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686\": container with ID starting with 93340de6afb03abba4d4d01084ef5ce6e22c7fa068b06b98bdb99464c847c686 not found: ID does not exist"
Sep 11 14:08:44.955717 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.955675    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"]
Sep 11 14:08:44.985216 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:44.985187    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-383709f1-65697f5bd6-plcjf"]
Sep 11 14:08:45.067712 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:45.067675    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"]
Sep 11 14:08:45.078355 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:45.078328    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-383709f1-58d76669d5-mxtff"]
Sep 11 14:08:45.685605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:45.685565    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"]
Sep 11 14:08:46.116540 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:46.116504    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817ee4af-2e6a-4bda-b46f-7aca14b41c62" path="/var/lib/kubelet/pods/817ee4af-2e6a-4bda-b46f-7aca14b41c62/volumes"
Sep 11 14:08:46.116928 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:46.116829    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8529d3d2-00cf-4697-9225-eb40f7afb30e" path="/var/lib/kubelet/pods/8529d3d2-00cf-4697-9225-eb40f7afb30e/volumes"
Sep 11 14:08:46.234541 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:46.234506    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"]
Sep 11 14:08:46.234766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:46.234716    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy" containerID="cri-o://19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21" gracePeriod=30
Sep 11 14:08:51.004001 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.003963    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"]
Sep 11 14:08:51.478535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.478504    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:51.559865 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.559829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-envoy\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560072 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.559903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgp4l\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-kube-api-access-zgp4l\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560072 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.559951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-certs\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560168 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-socket\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560168 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560118    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-token\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560168 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560161    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-data\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560309 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-credential-socket\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560309 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:08:51.560309 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-podinfo\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560462 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istiod-ca-cert\") pod \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\" (UID: \"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d\") "
Sep 11 14:08:51.560462 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:08:51.560559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:08:51.560619 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560595    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.560673 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560617    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-workload-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.560724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:08:51.560724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.560701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:08:51.562217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.562192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:08:51.562347 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.562286    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:08:51.562518 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.562493    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:08:51.562518 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.562505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-kube-api-access-zgp4l" (OuterVolumeSpecName: "kube-api-access-zgp4l") pod "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" (UID: "a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d"). InnerVolumeSpecName "kube-api-access-zgp4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661188    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661213    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-data\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661223    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-credential-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661232    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-podinfo\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661240    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istiod-ca-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661248    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-istio-envoy\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.661268 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.661256    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgp4l\" (UniqueName: \"kubernetes.io/projected/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d-kube-api-access-zgp4l\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:08:51.881668 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.881631    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerID="19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21" exitCode=0
Sep 11 14:08:51.881830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.881676    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"
Sep 11 14:08:51.881830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.881680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" event={"ID":"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d","Type":"ContainerDied","Data":"19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21"}
Sep 11 14:08:51.881830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.881761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6" event={"ID":"a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d","Type":"ContainerDied","Data":"9b82b448562d8b21aeb7f1ee8c91982bf3708d8bb24e0a6b6497672de3ef531d"}
Sep 11 14:08:51.881830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.881786    2561 scope.go:117] "RemoveContainer" containerID="19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21"
Sep 11 14:08:51.891071 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.891053    2561 scope.go:117] "RemoveContainer" containerID="de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"
Sep 11 14:08:51.898939 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.898904    2561 scope.go:117] "RemoveContainer" containerID="19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21"
Sep 11 14:08:51.899228 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:08:51.899206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21\": container with ID starting with 19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21 not found: ID does not exist" containerID="19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21"
Sep 11 14:08:51.899302 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.899237    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21"} err="failed to get container status \"19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21\": rpc error: code = NotFound desc = could not find container \"19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21\": container with ID starting with 19eea3f6c2cb15a20157dde08c63ff31ee72d28d8281b258a92a40852223ec21 not found: ID does not exist"
Sep 11 14:08:51.899302 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.899256    2561 scope.go:117] "RemoveContainer" containerID="de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"
Sep 11 14:08:51.899546 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:08:51.899526    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b\": container with ID starting with de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b not found: ID does not exist" containerID="de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"
Sep 11 14:08:51.899594 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.899552    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b"} err="failed to get container status \"de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b\": rpc error: code = NotFound desc = could not find container \"de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b\": container with ID starting with de2bf23f315bf9663d810c797462caa4a1d1f99281fe963e64f6b4cdeee09f0b not found: ID does not exist"
Sep 11 14:08:51.912627 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.912551    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"]
Sep 11 14:08:51.917282 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:51.917250    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-383709f1-openshift-default-59d9b996bb-4jqz6"]
Sep 11 14:08:52.117236 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.117199    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" path="/var/lib/kubelet/pods/a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d/volumes"
Sep 11 14:08:52.436366 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436333    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"]
Sep 11 14:08:52.436640 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436626    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy"
Sep 11 14:08:52.436640 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436641    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy"
Sep 11 14:08:52.436719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436662    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8529d3d2-00cf-4697-9225-eb40f7afb30e" containerName="payload-pre-processing"
Sep 11 14:08:52.436719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436668    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8529d3d2-00cf-4697-9225-eb40f7afb30e" containerName="payload-pre-processing"
Sep 11 14:08:52.436719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436677    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817ee4af-2e6a-4bda-b46f-7aca14b41c62" containerName="payload-processing"
Sep 11 14:08:52.436719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436683    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="817ee4af-2e6a-4bda-b46f-7aca14b41c62" containerName="payload-processing"
Sep 11 14:08:52.436858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436726    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy"
Sep 11 14:08:52.436858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436736    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8529d3d2-00cf-4697-9225-eb40f7afb30e" containerName="payload-pre-processing"
Sep 11 14:08:52.436858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436743    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy"
Sep 11 14:08:52.436858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.436749    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="817ee4af-2e6a-4bda-b46f-7aca14b41c62" containerName="payload-processing"
Sep 11 14:08:52.440799 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.440778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.445964 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.445943    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-61f2ffc9\""
Sep 11 14:08:52.446382 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.446365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-61f2ffc9-dockercfg-f2mph\""
Sep 11 14:08:52.460933 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.460889    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"]
Sep 11 14:08:52.533868 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.533837    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"]
Sep 11 14:08:52.534148 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.534136    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy"
Sep 11 14:08:52.534193 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.534151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c7cebc-1d51-4ce8-a8ee-fb353ff8168d" containerName="istio-proxy"
Sep 11 14:08:52.536983 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.536967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.549571 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.549549    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"]
Sep 11 14:08:52.567859 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.567821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4066774-b771-41e0-b4c9-820e8e222557-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.568056 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.567919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lst4p\" (UniqueName: \"kubernetes.io/projected/a4066774-b771-41e0-b4c9-820e8e222557-kube-api-access-lst4p\") pod \"payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.668749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.668718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lst4p\" (UniqueName: \"kubernetes.io/projected/a4066774-b771-41e0-b4c9-820e8e222557-kube-api-access-lst4p\") pod \"payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.668951 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.668762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960f1a09-ebfc-4305-a995-8181dec1630d-plugins-config-volume\") pod \"payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") " pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.668951 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.668808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdvd\" (UniqueName: \"kubernetes.io/projected/960f1a09-ebfc-4305-a995-8181dec1630d-kube-api-access-kpdvd\") pod \"payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") " pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.668951 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.668870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4066774-b771-41e0-b4c9-820e8e222557-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.669598 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.669580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4066774-b771-41e0-b4c9-820e8e222557-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.702731 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.702655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lst4p\" (UniqueName: \"kubernetes.io/projected/a4066774-b771-41e0-b4c9-820e8e222557-kube-api-access-lst4p\") pod \"payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.750400 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.750364    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:52.770015 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.769985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960f1a09-ebfc-4305-a995-8181dec1630d-plugins-config-volume\") pod \"payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") " pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.770151 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.770033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdvd\" (UniqueName: \"kubernetes.io/projected/960f1a09-ebfc-4305-a995-8181dec1630d-kube-api-access-kpdvd\") pod \"payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") " pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.770887 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.770862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960f1a09-ebfc-4305-a995-8181dec1630d-plugins-config-volume\") pod \"payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") " pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.786123 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.786086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdvd\" (UniqueName: \"kubernetes.io/projected/960f1a09-ebfc-4305-a995-8181dec1630d-kube-api-access-kpdvd\") pod \"payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") " pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.847370 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.847329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:52.853517 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:52.853488    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:08:53.028582 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.028554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"]
Sep 11 14:08:53.030825 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:08:53.030793    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4066774_b771_41e0_b4c9_820e8e222557.slice/crio-7f7671af6a645c0d3cf6c697fd5b0da4bd7f4224ffdfa428485869f5940635a5 WatchSource:0}: Error finding container 7f7671af6a645c0d3cf6c697fd5b0da4bd7f4224ffdfa428485869f5940635a5: Status 404 returned error can't find the container with id 7f7671af6a645c0d3cf6c697fd5b0da4bd7f4224ffdfa428485869f5940635a5
Sep 11 14:08:53.040327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.040302    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:53.040429 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.040347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:53.054548 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.054526    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:53.138191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.138160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"]
Sep 11 14:08:53.141166 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:08:53.141141    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960f1a09_ebfc_4305_a995_8181dec1630d.slice/crio-370e0dec06295959cee1d5acd2adb95fc27633c936affa84affdce40a9486dd2 WatchSource:0}: Error finding container 370e0dec06295959cee1d5acd2adb95fc27633c936affa84affdce40a9486dd2: Status 404 returned error can't find the container with id 370e0dec06295959cee1d5acd2adb95fc27633c936affa84affdce40a9486dd2
Sep 11 14:08:53.894190 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.894136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf" event={"ID":"a4066774-b771-41e0-b4c9-820e8e222557","Type":"ContainerStarted","Data":"e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb"}
Sep 11 14:08:53.894379 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.894244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:08:53.894379 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.894267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf" event={"ID":"a4066774-b771-41e0-b4c9-820e8e222557","Type":"ContainerStarted","Data":"7f7671af6a645c0d3cf6c697fd5b0da4bd7f4224ffdfa428485869f5940635a5"}
Sep 11 14:08:53.895766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.895736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586" event={"ID":"960f1a09-ebfc-4305-a995-8181dec1630d","Type":"ContainerStarted","Data":"c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3"}
Sep 11 14:08:53.895897 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.895794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586" event={"ID":"960f1a09-ebfc-4305-a995-8181dec1630d","Type":"ContainerStarted","Data":"370e0dec06295959cee1d5acd2adb95fc27633c936affa84affdce40a9486dd2"}
Sep 11 14:08:53.895897 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.895867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:08:53.907345 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.907310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:08:53.915644 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.915599    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf" podStartSLOduration=1.915586607 podStartE2EDuration="1.915586607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:53.914221521 +0000 UTC m=+982.281810965" watchObservedRunningTime="2026-09-11 14:08:53.915586607 +0000 UTC m=+982.283176050"
Sep 11 14:08:53.936535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:08:53.936476    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586" podStartSLOduration=1.936455779 podStartE2EDuration="1.936455779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:53.936049442 +0000 UTC m=+982.303638885" watchObservedRunningTime="2026-09-11 14:08:53.936455779 +0000 UTC m=+982.304045222"
Sep 11 14:09:00.704418 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:00.704352    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l" podUID="a74e6bd3-cd3a-4552-bb00-65bc829fb16b" containerName="main" containerID="cri-o://6dce8a19513e8d1759e57ce33ee4fd931483daaa5a5e38109620f1ff85ae8457" gracePeriod=45
Sep 11 14:09:00.928789 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:00.928760    2561 generic.go:356] "Generic (PLEG): container finished" podID="a74e6bd3-cd3a-4552-bb00-65bc829fb16b" containerID="6dce8a19513e8d1759e57ce33ee4fd931483daaa5a5e38109620f1ff85ae8457" exitCode=0
Sep 11 14:09:00.928884 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:00.928830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l" event={"ID":"a74e6bd3-cd3a-4552-bb00-65bc829fb16b","Type":"ContainerDied","Data":"6dce8a19513e8d1759e57ce33ee4fd931483daaa5a5e38109620f1ff85ae8457"}
Sep 11 14:09:00.947470 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:00.947445    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:09:01.144635 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tmp-dir\") pod \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") "
Sep 11 14:09:01.144635 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144634    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-dshm\") pod \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") "
Sep 11 14:09:01.144898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144664    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-model-cache\") pod \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") "
Sep 11 14:09:01.144898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144726    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ddtq\" (UniqueName: \"kubernetes.io/projected/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-kube-api-access-2ddtq\") pod \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") "
Sep 11 14:09:01.144898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144747    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-home\") pod \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") "
Sep 11 14:09:01.144898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144806    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tls-certs\") pod \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\" (UID: \"a74e6bd3-cd3a-4552-bb00-65bc829fb16b\") "
Sep 11 14:09:01.145131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.144959    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a74e6bd3-cd3a-4552-bb00-65bc829fb16b" (UID: "a74e6bd3-cd3a-4552-bb00-65bc829fb16b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:01.145131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.145061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a74e6bd3-cd3a-4552-bb00-65bc829fb16b" (UID: "a74e6bd3-cd3a-4552-bb00-65bc829fb16b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:01.145131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.145076    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-home" (OuterVolumeSpecName: "home") pod "a74e6bd3-cd3a-4552-bb00-65bc829fb16b" (UID: "a74e6bd3-cd3a-4552-bb00-65bc829fb16b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:01.145287 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.145145    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:01.145287 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.145164    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:01.147035 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.147010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-dshm" (OuterVolumeSpecName: "dshm") pod "a74e6bd3-cd3a-4552-bb00-65bc829fb16b" (UID: "a74e6bd3-cd3a-4552-bb00-65bc829fb16b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:01.147138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.147027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-kube-api-access-2ddtq" (OuterVolumeSpecName: "kube-api-access-2ddtq") pod "a74e6bd3-cd3a-4552-bb00-65bc829fb16b" (UID: "a74e6bd3-cd3a-4552-bb00-65bc829fb16b"). InnerVolumeSpecName "kube-api-access-2ddtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:09:01.147138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.147010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a74e6bd3-cd3a-4552-bb00-65bc829fb16b" (UID: "a74e6bd3-cd3a-4552-bb00-65bc829fb16b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:09:01.246065 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.246035    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:01.246065 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.246063    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddtq\" (UniqueName: \"kubernetes.io/projected/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-kube-api-access-2ddtq\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:01.246065 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.246074    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:01.246293 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.246084    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74e6bd3-cd3a-4552-bb00-65bc829fb16b-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:01.935295 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.935255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l" event={"ID":"a74e6bd3-cd3a-4552-bb00-65bc829fb16b","Type":"ContainerDied","Data":"a33ccb36d494695d17e0cf865338338924bbf55379507eb5222fb3ffb02b82fb"}
Sep 11 14:09:01.935295 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.935271    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"
Sep 11 14:09:01.935768 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.935302    2561 scope.go:117] "RemoveContainer" containerID="6dce8a19513e8d1759e57ce33ee4fd931483daaa5a5e38109620f1ff85ae8457"
Sep 11 14:09:01.959103 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.959079    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"]
Sep 11 14:09:01.964185 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:01.964162    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-918362dd-kserve-5c6b96cc6-tmd9l"]
Sep 11 14:09:02.117947 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:02.117885    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74e6bd3-cd3a-4552-bb00-65bc829fb16b" path="/var/lib/kubelet/pods/a74e6bd3-cd3a-4552-bb00-65bc829fb16b/volumes"
Sep 11 14:09:04.018936 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.018353    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"]
Sep 11 14:09:04.018936 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.018667    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g" podUID="66828599-9e10-4384-be49-a72e1cf11caa" containerName="payload-processing" containerID="cri-o://c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d" gracePeriod=30
Sep 11 14:09:04.273260 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.273199    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:09:04.369479 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.369449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66828599-9e10-4384-be49-a72e1cf11caa-plugins-config-volume\") pod \"66828599-9e10-4384-be49-a72e1cf11caa\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") "
Sep 11 14:09:04.369639 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.369490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctzxp\" (UniqueName: \"kubernetes.io/projected/66828599-9e10-4384-be49-a72e1cf11caa-kube-api-access-ctzxp\") pod \"66828599-9e10-4384-be49-a72e1cf11caa\" (UID: \"66828599-9e10-4384-be49-a72e1cf11caa\") "
Sep 11 14:09:04.369802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.369779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66828599-9e10-4384-be49-a72e1cf11caa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66828599-9e10-4384-be49-a72e1cf11caa" (UID: "66828599-9e10-4384-be49-a72e1cf11caa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:09:04.371504 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.371480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66828599-9e10-4384-be49-a72e1cf11caa-kube-api-access-ctzxp" (OuterVolumeSpecName: "kube-api-access-ctzxp") pod "66828599-9e10-4384-be49-a72e1cf11caa" (UID: "66828599-9e10-4384-be49-a72e1cf11caa"). InnerVolumeSpecName "kube-api-access-ctzxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:09:04.469992 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.469957    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66828599-9e10-4384-be49-a72e1cf11caa-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:04.469992 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.469986    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctzxp\" (UniqueName: \"kubernetes.io/projected/66828599-9e10-4384-be49-a72e1cf11caa-kube-api-access-ctzxp\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:04.902393 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.902358    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:09:04.902611 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.902429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:09:04.951228 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.951191    2561 generic.go:356] "Generic (PLEG): container finished" podID="66828599-9e10-4384-be49-a72e1cf11caa" containerID="c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d" exitCode=0
Sep 11 14:09:04.951228 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.951228    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"
Sep 11 14:09:04.951428 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.951269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g" event={"ID":"66828599-9e10-4384-be49-a72e1cf11caa","Type":"ContainerDied","Data":"c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d"}
Sep 11 14:09:04.951428 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.951321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g" event={"ID":"66828599-9e10-4384-be49-a72e1cf11caa","Type":"ContainerDied","Data":"80222956430b14ca3f72fb242c19c6a2a932a43bcf8179b29646e11fcc293c24"}
Sep 11 14:09:04.951428 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.951349    2561 scope.go:117] "RemoveContainer" containerID="c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d"
Sep 11 14:09:04.961490 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.961471    2561 scope.go:117] "RemoveContainer" containerID="c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d"
Sep 11 14:09:04.961815 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:09:04.961796    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d\": container with ID starting with c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d not found: ID does not exist" containerID="c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d"
Sep 11 14:09:04.961860 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.961824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d"} err="failed to get container status \"c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d\": rpc error: code = NotFound desc = could not find container \"c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d\": container with ID starting with c30fbe0e0e4a3112f20aa36ebfbb0ae3457f940caa6703f71fbd3194ff69ca7d not found: ID does not exist"
Sep 11 14:09:04.989834 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.989802    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"]
Sep 11 14:09:04.999041 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:04.999015    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0bd77b-6c464bbcbf-7bk4g"]
Sep 11 14:09:06.022212 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.022165    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2" podUID="f1486892-5bb1-4d19-8063-14005481cef7" containerName="main" containerID="cri-o://1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224" gracePeriod=45
Sep 11 14:09:06.116368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.116334    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66828599-9e10-4384-be49-a72e1cf11caa" path="/var/lib/kubelet/pods/66828599-9e10-4384-be49-a72e1cf11caa/volumes"
Sep 11 14:09:06.267350 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.267323    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:09:06.282956 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.282871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-dshm\") pod \"f1486892-5bb1-4d19-8063-14005481cef7\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") "
Sep 11 14:09:06.282956 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.282940    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1486892-5bb1-4d19-8063-14005481cef7-tls-certs\") pod \"f1486892-5bb1-4d19-8063-14005481cef7\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") "
Sep 11 14:09:06.283140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.282968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-model-cache\") pod \"f1486892-5bb1-4d19-8063-14005481cef7\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") "
Sep 11 14:09:06.283140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.282996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq8vm\" (UniqueName: \"kubernetes.io/projected/f1486892-5bb1-4d19-8063-14005481cef7-kube-api-access-lq8vm\") pod \"f1486892-5bb1-4d19-8063-14005481cef7\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") "
Sep 11 14:09:06.283140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-home\") pod \"f1486892-5bb1-4d19-8063-14005481cef7\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") "
Sep 11 14:09:06.283291 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-tmp-dir\") pod \"f1486892-5bb1-4d19-8063-14005481cef7\" (UID: \"f1486892-5bb1-4d19-8063-14005481cef7\") "
Sep 11 14:09:06.283291 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1486892-5bb1-4d19-8063-14005481cef7" (UID: "f1486892-5bb1-4d19-8063-14005481cef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:06.283468 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-home" (OuterVolumeSpecName: "home") pod "f1486892-5bb1-4d19-8063-14005481cef7" (UID: "f1486892-5bb1-4d19-8063-14005481cef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:06.283468 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1486892-5bb1-4d19-8063-14005481cef7" (UID: "f1486892-5bb1-4d19-8063-14005481cef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:06.283468 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283445    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:06.283468 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.283457    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:06.285219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.285196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1486892-5bb1-4d19-8063-14005481cef7-kube-api-access-lq8vm" (OuterVolumeSpecName: "kube-api-access-lq8vm") pod "f1486892-5bb1-4d19-8063-14005481cef7" (UID: "f1486892-5bb1-4d19-8063-14005481cef7"). InnerVolumeSpecName "kube-api-access-lq8vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:09:06.285461 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.285440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-dshm" (OuterVolumeSpecName: "dshm") pod "f1486892-5bb1-4d19-8063-14005481cef7" (UID: "f1486892-5bb1-4d19-8063-14005481cef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:09:06.285503 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.285440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1486892-5bb1-4d19-8063-14005481cef7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1486892-5bb1-4d19-8063-14005481cef7" (UID: "f1486892-5bb1-4d19-8063-14005481cef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:09:06.384393 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.384362    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:06.384393 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.384387    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1486892-5bb1-4d19-8063-14005481cef7-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:06.384393 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.384397    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8vm\" (UniqueName: \"kubernetes.io/projected/f1486892-5bb1-4d19-8063-14005481cef7-kube-api-access-lq8vm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:06.384625 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.384406    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1486892-5bb1-4d19-8063-14005481cef7-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:06.962510 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.962476    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1486892-5bb1-4d19-8063-14005481cef7" containerID="1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224" exitCode=0
Sep 11 14:09:06.962704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.962519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2" event={"ID":"f1486892-5bb1-4d19-8063-14005481cef7","Type":"ContainerDied","Data":"1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224"}
Sep 11 14:09:06.962704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.962523    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"
Sep 11 14:09:06.962704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.962544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2" event={"ID":"f1486892-5bb1-4d19-8063-14005481cef7","Type":"ContainerDied","Data":"079ef36dd16510b7011c1715f084eba0d4e8fb9f4cfe2c08fffa2ec5d67ccb37"}
Sep 11 14:09:06.962704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.962561    2561 scope.go:117] "RemoveContainer" containerID="1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224"
Sep 11 14:09:06.971156 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.971136    2561 scope.go:117] "RemoveContainer" containerID="1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224"
Sep 11 14:09:06.971414 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:09:06.971395    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224\": container with ID starting with 1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224 not found: ID does not exist" containerID="1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224"
Sep 11 14:09:06.971465 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.971423    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224"} err="failed to get container status \"1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224\": rpc error: code = NotFound desc = could not find container \"1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224\": container with ID starting with 1c51c28acf69bc338627bc30de97017662b22a2f330f39e3af46bc3c1f2b8224 not found: ID does not exist"
Sep 11 14:09:06.992839 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:06.992810    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"]
Sep 11 14:09:07.032495 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:07.032464    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-918362dd-kserve-84c866b7fd-tjzl2"]
Sep 11 14:09:08.121699 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:08.121665    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1486892-5bb1-4d19-8063-14005481cef7" path="/var/lib/kubelet/pods/f1486892-5bb1-4d19-8063-14005481cef7/volumes"
Sep 11 14:09:09.192972 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.192943    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"]
Sep 11 14:09:09.193443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193362    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74e6bd3-cd3a-4552-bb00-65bc829fb16b" containerName="main"
Sep 11 14:09:09.193443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193380    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74e6bd3-cd3a-4552-bb00-65bc829fb16b" containerName="main"
Sep 11 14:09:09.193443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193396    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1486892-5bb1-4d19-8063-14005481cef7" containerName="main"
Sep 11 14:09:09.193443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193404    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1486892-5bb1-4d19-8063-14005481cef7" containerName="main"
Sep 11 14:09:09.193443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193417    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66828599-9e10-4384-be49-a72e1cf11caa" containerName="payload-processing"
Sep 11 14:09:09.193443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193426    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66828599-9e10-4384-be49-a72e1cf11caa" containerName="payload-processing"
Sep 11 14:09:09.193754 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74e6bd3-cd3a-4552-bb00-65bc829fb16b" containerName="main"
Sep 11 14:09:09.193754 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193524    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1486892-5bb1-4d19-8063-14005481cef7" containerName="main"
Sep 11 14:09:09.193754 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.193535    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66828599-9e10-4384-be49-a72e1cf11caa" containerName="payload-processing"
Sep 11 14:09:09.197568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.197548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:09.201290 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.201269    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:09:09.201400 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.201272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:09:09.207437 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.207415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bbf334-8e5f-4068-81fc-56628663fde4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp\" (UID: \"61bbf334-8e5f-4068-81fc-56628663fde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:09.208577 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.208552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"]
Sep 11 14:09:09.308653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.308621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bbf334-8e5f-4068-81fc-56628663fde4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp\" (UID: \"61bbf334-8e5f-4068-81fc-56628663fde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:09.309243 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.309224    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bbf334-8e5f-4068-81fc-56628663fde4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp\" (UID: \"61bbf334-8e5f-4068-81fc-56628663fde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:09.507627 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.507553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:09.778438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.778371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"]
Sep 11 14:09:09.781454 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:09:09.781430    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bbf334_8e5f_4068_81fc_56628663fde4.slice/crio-6bb4fd34e97f65b030b92ecc12ab1effca136fac83c517b7db7b3a2e6a3b65b5 WatchSource:0}: Error finding container 6bb4fd34e97f65b030b92ecc12ab1effca136fac83c517b7db7b3a2e6a3b65b5: Status 404 returned error can't find the container with id 6bb4fd34e97f65b030b92ecc12ab1effca136fac83c517b7db7b3a2e6a3b65b5
Sep 11 14:09:09.978401 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:09.978372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp" event={"ID":"61bbf334-8e5f-4068-81fc-56628663fde4","Type":"ContainerStarted","Data":"6bb4fd34e97f65b030b92ecc12ab1effca136fac83c517b7db7b3a2e6a3b65b5"}
Sep 11 14:09:10.809054 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:10.809018    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:09:10.984752 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:10.984717    2561 generic.go:356] "Generic (PLEG): container finished" podID="61bbf334-8e5f-4068-81fc-56628663fde4" containerID="a98708d7cd1eb4ad3d17881b0c47a78335c79ccd639ebea622b97299114bab41" exitCode=0
Sep 11 14:09:10.984930 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:10.984776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp" event={"ID":"61bbf334-8e5f-4068-81fc-56628663fde4","Type":"ContainerDied","Data":"a98708d7cd1eb4ad3d17881b0c47a78335c79ccd639ebea622b97299114bab41"}
Sep 11 14:09:12.113855 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.113830    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:12.130785 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.130762    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bbf334-8e5f-4068-81fc-56628663fde4-maas-api-service-ca\") pod \"61bbf334-8e5f-4068-81fc-56628663fde4\" (UID: \"61bbf334-8e5f-4068-81fc-56628663fde4\") "
Sep 11 14:09:12.131079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.131060    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bbf334-8e5f-4068-81fc-56628663fde4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61bbf334-8e5f-4068-81fc-56628663fde4" (UID: "61bbf334-8e5f-4068-81fc-56628663fde4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:09:12.232222 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.232192    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bbf334-8e5f-4068-81fc-56628663fde4-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:12.953403 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.953371    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"]
Sep 11 14:09:12.953697 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.953685    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61bbf334-8e5f-4068-81fc-56628663fde4" containerName="revoke-keys"
Sep 11 14:09:12.953764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.953700    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="61bbf334-8e5f-4068-81fc-56628663fde4" containerName="revoke-keys"
Sep 11 14:09:12.953764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.953753    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="61bbf334-8e5f-4068-81fc-56628663fde4" containerName="revoke-keys"
Sep 11 14:09:12.956494 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.956478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:12.959792 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.959768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-25f5d3-openshift-default-dockercfg-jzzzk\""
Sep 11 14:09:12.959943 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.959851    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:09:12.974653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.974629    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"]
Sep 11 14:09:12.994068 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.994042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp" event={"ID":"61bbf334-8e5f-4068-81fc-56628663fde4","Type":"ContainerDied","Data":"6bb4fd34e97f65b030b92ecc12ab1effca136fac83c517b7db7b3a2e6a3b65b5"}
Sep 11 14:09:12.994068 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.994069    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"
Sep 11 14:09:12.994223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:12.994084    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb4fd34e97f65b030b92ecc12ab1effca136fac83c517b7db7b3a2e6a3b65b5"
Sep 11 14:09:13.038691 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istiod-ca-cert\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038691 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-envoy\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-socket\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lkh\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-kube-api-access-r6lkh\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-token\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-credential-socket\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038992 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-certs\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.038992 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.038927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-podinfo\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.039049 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.039000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-data\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140316 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-podinfo\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-data\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istiod-ca-cert\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-envoy\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-socket\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lkh\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-kube-api-access-r6lkh\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-token\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-credential-socket\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.140695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-certs\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.141079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-socket\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.141079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.140963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-certs\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.141079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.141029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-credential-socket\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.141191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.141095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-data\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.141225 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.141191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istiod-ca-cert\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.143240 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.143216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-envoy\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.143551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.143530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-podinfo\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.155692 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.155668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lkh\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-kube-api-access-r6lkh\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.156591 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.156566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-token\") pod \"e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") " pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.266443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.266361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:13.562034 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:09:13.562005    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abd9a11_e015_48f6_a4f5_6657a03e2c6a.slice/crio-c17a367be01bcd4d34f32c125100d7064c79128063c5b5448108169bb51cc1b2 WatchSource:0}: Error finding container c17a367be01bcd4d34f32c125100d7064c79128063c5b5448108169bb51cc1b2: Status 404 returned error can't find the container with id c17a367be01bcd4d34f32c125100d7064c79128063c5b5448108169bb51cc1b2
Sep 11 14:09:13.563232 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.563208    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"]
Sep 11 14:09:13.564404 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.564374    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:09:13.564469 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.564437    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:09:13.564469 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.564465    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:09:13.999921 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.999815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" event={"ID":"2abd9a11-e015-48f6-a4f5-6657a03e2c6a","Type":"ContainerStarted","Data":"de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"}
Sep 11 14:09:13.999921 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:13.999872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" event={"ID":"2abd9a11-e015-48f6-a4f5-6657a03e2c6a","Type":"ContainerStarted","Data":"c17a367be01bcd4d34f32c125100d7064c79128063c5b5448108169bb51cc1b2"}
Sep 11 14:09:14.063802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:14.063749    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podStartSLOduration=2.063733218 podStartE2EDuration="2.063733218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:09:14.062904287 +0000 UTC m=+1002.430493729" watchObservedRunningTime="2026-09-11 14:09:14.063733218 +0000 UTC m=+1002.431322660"
Sep 11 14:09:14.267081 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:14.267001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:14.272142 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:14.272116    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:15.004168 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:15.004136    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:15.005097 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:15.005080    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:19.876991 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.876961    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"]
Sep 11 14:09:19.880394 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.880377    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:19.883113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.883089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-25f5d3-dockercfg-tfc65\""
Sep 11 14:09:19.884572 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.884555    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-25f5d3\""
Sep 11 14:09:19.891417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.891395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"]
Sep 11 14:09:19.995011 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.994980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzwm\" (UniqueName: \"kubernetes.io/projected/c31f3169-bd75-4e3b-9b73-900e1be15c50-kube-api-access-gtzwm\") pod \"payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") " pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:19.995156 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:19.995097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f3169-bd75-4e3b-9b73-900e1be15c50-plugins-config-volume\") pod \"payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") " pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:20.096429 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.096401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzwm\" (UniqueName: \"kubernetes.io/projected/c31f3169-bd75-4e3b-9b73-900e1be15c50-kube-api-access-gtzwm\") pod \"payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") " pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:20.096576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.096472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f3169-bd75-4e3b-9b73-900e1be15c50-plugins-config-volume\") pod \"payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") " pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:20.097099 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.097081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f3169-bd75-4e3b-9b73-900e1be15c50-plugins-config-volume\") pod \"payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") " pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:20.106440 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.106409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzwm\" (UniqueName: \"kubernetes.io/projected/c31f3169-bd75-4e3b-9b73-900e1be15c50-kube-api-access-gtzwm\") pod \"payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") " pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:20.173827 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.173749    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"]
Sep 11 14:09:20.174003 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.173966    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586" podUID="960f1a09-ebfc-4305-a995-8181dec1630d" containerName="payload-processing" containerID="cri-o://c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3" gracePeriod=30
Sep 11 14:09:20.181903 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.181880    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"]
Sep 11 14:09:20.182104 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.182084    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf" podUID="a4066774-b771-41e0-b4c9-820e8e222557" containerName="payload-pre-processing" containerID="cri-o://e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb" gracePeriod=30
Sep 11 14:09:20.191874 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.191850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:20.374584 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.374550    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"]
Sep 11 14:09:20.377792 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.377768    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-61f2ffc9-49gdp"]
Sep 11 14:09:20.424387 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.424327    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:09:20.431338 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.431317    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:09:20.477122 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.476943    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"]
Sep 11 14:09:20.482745 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:09:20.482703    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31f3169_bd75_4e3b_9b73_900e1be15c50.slice/crio-7dfa4d9f5c3b7f5f68597a3f92434ded901ed6717114127f0bfdadf4365ac52a WatchSource:0}: Error finding container 7dfa4d9f5c3b7f5f68597a3f92434ded901ed6717114127f0bfdadf4365ac52a: Status 404 returned error can't find the container with id 7dfa4d9f5c3b7f5f68597a3f92434ded901ed6717114127f0bfdadf4365ac52a
Sep 11 14:09:20.500491 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.500470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lst4p\" (UniqueName: \"kubernetes.io/projected/a4066774-b771-41e0-b4c9-820e8e222557-kube-api-access-lst4p\") pod \"a4066774-b771-41e0-b4c9-820e8e222557\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") "
Sep 11 14:09:20.500598 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.500534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960f1a09-ebfc-4305-a995-8181dec1630d-plugins-config-volume\") pod \"960f1a09-ebfc-4305-a995-8181dec1630d\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") "
Sep 11 14:09:20.500598 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.500566    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpdvd\" (UniqueName: \"kubernetes.io/projected/960f1a09-ebfc-4305-a995-8181dec1630d-kube-api-access-kpdvd\") pod \"960f1a09-ebfc-4305-a995-8181dec1630d\" (UID: \"960f1a09-ebfc-4305-a995-8181dec1630d\") "
Sep 11 14:09:20.500735 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.500597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4066774-b771-41e0-b4c9-820e8e222557-plugins-config-volume\") pod \"a4066774-b771-41e0-b4c9-820e8e222557\" (UID: \"a4066774-b771-41e0-b4c9-820e8e222557\") "
Sep 11 14:09:20.501204 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.500962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960f1a09-ebfc-4305-a995-8181dec1630d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "960f1a09-ebfc-4305-a995-8181dec1630d" (UID: "960f1a09-ebfc-4305-a995-8181dec1630d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:09:20.501204 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.501136    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4066774-b771-41e0-b4c9-820e8e222557-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4066774-b771-41e0-b4c9-820e8e222557" (UID: "a4066774-b771-41e0-b4c9-820e8e222557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:09:20.505046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.505023    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960f1a09-ebfc-4305-a995-8181dec1630d-kube-api-access-kpdvd" (OuterVolumeSpecName: "kube-api-access-kpdvd") pod "960f1a09-ebfc-4305-a995-8181dec1630d" (UID: "960f1a09-ebfc-4305-a995-8181dec1630d"). InnerVolumeSpecName "kube-api-access-kpdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:09:20.507287 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.507262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4066774-b771-41e0-b4c9-820e8e222557-kube-api-access-lst4p" (OuterVolumeSpecName: "kube-api-access-lst4p") pod "a4066774-b771-41e0-b4c9-820e8e222557" (UID: "a4066774-b771-41e0-b4c9-820e8e222557"). InnerVolumeSpecName "kube-api-access-lst4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:09:20.601666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.601626    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lst4p\" (UniqueName: \"kubernetes.io/projected/a4066774-b771-41e0-b4c9-820e8e222557-kube-api-access-lst4p\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:20.601666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.601653    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960f1a09-ebfc-4305-a995-8181dec1630d-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:20.601666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.601665    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdvd\" (UniqueName: \"kubernetes.io/projected/960f1a09-ebfc-4305-a995-8181dec1630d-kube-api-access-kpdvd\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:20.601666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:20.601674    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4066774-b771-41e0-b4c9-820e8e222557-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:09:21.031337 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.031304    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4066774-b771-41e0-b4c9-820e8e222557" containerID="e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb" exitCode=0
Sep 11 14:09:21.031795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.031349    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"
Sep 11 14:09:21.031795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.031389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf" event={"ID":"a4066774-b771-41e0-b4c9-820e8e222557","Type":"ContainerDied","Data":"e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb"}
Sep 11 14:09:21.031795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.031436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf" event={"ID":"a4066774-b771-41e0-b4c9-820e8e222557","Type":"ContainerDied","Data":"7f7671af6a645c0d3cf6c697fd5b0da4bd7f4224ffdfa428485869f5940635a5"}
Sep 11 14:09:21.031795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.031461    2561 scope.go:117] "RemoveContainer" containerID="e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb"
Sep 11 14:09:21.032826 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.032799    2561 generic.go:356] "Generic (PLEG): container finished" podID="960f1a09-ebfc-4305-a995-8181dec1630d" containerID="c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3" exitCode=0
Sep 11 14:09:21.032974 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.032830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586" event={"ID":"960f1a09-ebfc-4305-a995-8181dec1630d","Type":"ContainerDied","Data":"c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3"}
Sep 11 14:09:21.032974 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.032847    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"
Sep 11 14:09:21.032974 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.032879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586" event={"ID":"960f1a09-ebfc-4305-a995-8181dec1630d","Type":"ContainerDied","Data":"370e0dec06295959cee1d5acd2adb95fc27633c936affa84affdce40a9486dd2"}
Sep 11 14:09:21.034743 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.034720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6" event={"ID":"c31f3169-bd75-4e3b-9b73-900e1be15c50","Type":"ContainerStarted","Data":"6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f"}
Sep 11 14:09:21.034864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.034791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6" event={"ID":"c31f3169-bd75-4e3b-9b73-900e1be15c50","Type":"ContainerStarted","Data":"7dfa4d9f5c3b7f5f68597a3f92434ded901ed6717114127f0bfdadf4365ac52a"}
Sep 11 14:09:21.034864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.034847    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:21.040799 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.040487    2561 scope.go:117] "RemoveContainer" containerID="e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb"
Sep 11 14:09:21.040868 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:09:21.040850    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb\": container with ID starting with e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb not found: ID does not exist" containerID="e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb"
Sep 11 14:09:21.040947 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.040879    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb"} err="failed to get container status \"e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb\": rpc error: code = NotFound desc = could not find container \"e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb\": container with ID starting with e8a116e6a44dd9787357dc285d7fa3ff42142e3ec009da743a7777a186bb6beb not found: ID does not exist"
Sep 11 14:09:21.040947 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.040897    2561 scope.go:117] "RemoveContainer" containerID="c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3"
Sep 11 14:09:21.048333 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.048316    2561 scope.go:117] "RemoveContainer" containerID="c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3"
Sep 11 14:09:21.048565 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:09:21.048548    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3\": container with ID starting with c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3 not found: ID does not exist" containerID="c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3"
Sep 11 14:09:21.048622 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.048570    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3"} err="failed to get container status \"c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3\": rpc error: code = NotFound desc = could not find container \"c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3\": container with ID starting with c9f83aacb4c7b5391b764acf70f1cc2b1d1c5e2ecf94f7b3346b04a307c6a9b3 not found: ID does not exist"
Sep 11 14:09:21.061475 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.061430    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6" podStartSLOduration=2.061413637 podStartE2EDuration="2.061413637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:09:21.060984478 +0000 UTC m=+1009.428573921" watchObservedRunningTime="2026-09-11 14:09:21.061413637 +0000 UTC m=+1009.429003080"
Sep 11 14:09:21.077454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.073779    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"]
Sep 11 14:09:21.080004 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.079977    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-61f2ffc9-8688ddbfdb-gfdqf"]
Sep 11 14:09:21.094400 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.094377    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"]
Sep 11 14:09:21.100495 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:21.100475    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-61f2ffc9-6f846b66d9-zz586"]
Sep 11 14:09:22.116071 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:22.116038    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61bbf334-8e5f-4068-81fc-56628663fde4" path="/var/lib/kubelet/pods/61bbf334-8e5f-4068-81fc-56628663fde4/volumes"
Sep 11 14:09:22.116444 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:22.116366    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="960f1a09-ebfc-4305-a995-8181dec1630d" path="/var/lib/kubelet/pods/960f1a09-ebfc-4305-a995-8181dec1630d/volumes"
Sep 11 14:09:22.116672 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:22.116659    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4066774-b771-41e0-b4c9-820e8e222557" path="/var/lib/kubelet/pods/a4066774-b771-41e0-b4c9-820e8e222557/volumes"
Sep 11 14:09:23.484997 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.484963    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"]
Sep 11 14:09:23.485368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.485245    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4066774-b771-41e0-b4c9-820e8e222557" containerName="payload-pre-processing"
Sep 11 14:09:23.485368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.485257    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4066774-b771-41e0-b4c9-820e8e222557" containerName="payload-pre-processing"
Sep 11 14:09:23.485368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.485280    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960f1a09-ebfc-4305-a995-8181dec1630d" containerName="payload-processing"
Sep 11 14:09:23.485368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.485286    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="960f1a09-ebfc-4305-a995-8181dec1630d" containerName="payload-processing"
Sep 11 14:09:23.485368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.485335    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="960f1a09-ebfc-4305-a995-8181dec1630d" containerName="payload-processing"
Sep 11 14:09:23.485368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.485342    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4066774-b771-41e0-b4c9-820e8e222557" containerName="payload-pre-processing"
Sep 11 14:09:23.489658 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.489639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.492217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.492191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-80fb45\"/\"test-model-80fb45-kserve-self-signed-certs\""
Sep 11 14:09:23.493208 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.493192    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-80fb45\"/\"kube-root-ca.crt\""
Sep 11 14:09:23.493340 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.493323    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-80fb45\"/\"openshift-service-ca.crt\""
Sep 11 14:09:23.493389 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.493375    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-80fb45\"/\"default-dockercfg-ls7f6\""
Sep 11 14:09:23.503034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.503015    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"]
Sep 11 14:09:23.529897 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.529870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-model-cache\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.530017 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.529962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-dshm\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.530017 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.530004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-home\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.530104 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.530049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-tmp-dir\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.530104 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.530066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6znf\" (UniqueName: \"kubernetes.io/projected/3d581bf4-c624-459a-a680-e44fc4127df8-kube-api-access-g6znf\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.530185 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.530126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d581bf4-c624-459a-a680-e44fc4127df8-tls-certs\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631193 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-tmp-dir\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631193 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6znf\" (UniqueName: \"kubernetes.io/projected/3d581bf4-c624-459a-a680-e44fc4127df8-kube-api-access-g6znf\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d581bf4-c624-459a-a680-e44fc4127df8-tls-certs\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-model-cache\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-dshm\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-home\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631634 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-tmp-dir\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631686 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-model-cache\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.631765 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.631748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-home\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.633649 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.633621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-dshm\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.633920 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.633885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d581bf4-c624-459a-a680-e44fc4127df8-tls-certs\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.642241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.642219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6znf\" (UniqueName: \"kubernetes.io/projected/3d581bf4-c624-459a-a680-e44fc4127df8-kube-api-access-g6znf\") pod \"test-model-80fb45-kserve-77b44886bc-g6f2t\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") " pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:23.798747 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:23.798662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:24.076100 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:24.076070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"]
Sep 11 14:09:24.079068 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:09:24.079040    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d581bf4_c624_459a_a680_e44fc4127df8.slice/crio-707f850c6fd67403931491afd0e014195cc0015ffa74d8e95efc62a0dd5f4b4f WatchSource:0}: Error finding container 707f850c6fd67403931491afd0e014195cc0015ffa74d8e95efc62a0dd5f4b4f: Status 404 returned error can't find the container with id 707f850c6fd67403931491afd0e014195cc0015ffa74d8e95efc62a0dd5f4b4f
Sep 11 14:09:25.057576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:25.057540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t" event={"ID":"3d581bf4-c624-459a-a680-e44fc4127df8","Type":"ContainerStarted","Data":"48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc"}
Sep 11 14:09:25.057576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:25.057582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t" event={"ID":"3d581bf4-c624-459a-a680-e44fc4127df8","Type":"ContainerStarted","Data":"707f850c6fd67403931491afd0e014195cc0015ffa74d8e95efc62a0dd5f4b4f"}
Sep 11 14:09:28.029807 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:28.029739    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t" podStartSLOduration=5.029713839 podStartE2EDuration="5.029713839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:09:25.08528397 +0000 UTC m=+1013.452873413" watchObservedRunningTime="2026-09-11 14:09:28.029713839 +0000 UTC m=+1016.397303313"
Sep 11 14:09:28.030223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:28.030053    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:09:32.044044 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:32.044002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:09:33.799606 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:33.799567    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:33.800016 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:33.799656    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:33.812069 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:33.812039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:34.117477 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:34.117448    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:09:38.123243 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:38.123145    2561 generic.go:356] "Generic (PLEG): container finished" podID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerID="de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a" exitCode=0
Sep 11 14:09:38.123243 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:38.123219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" event={"ID":"2abd9a11-e015-48f6-a4f5-6657a03e2c6a","Type":"ContainerDied","Data":"de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"}
Sep 11 14:09:38.123761 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:38.123586    2561 scope.go:117] "RemoveContainer" containerID="de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"
Sep 11 14:09:38.124776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:38.124744    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:09:38.124852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:38.124836    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:09:38.124903 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:38.124887    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:09:39.132692 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:39.132644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" event={"ID":"2abd9a11-e015-48f6-a4f5-6657a03e2c6a","Type":"ContainerStarted","Data":"5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa"}
Sep 11 14:09:39.266503 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:39.266459    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:39.266926 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:39.266875    2561 patch_prober.go:28] interesting pod/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 11 14:09:39.267017 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:39.266951    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 11 14:09:40.267278 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:40.267240    2561 patch_prober.go:28] interesting pod/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 11 14:09:40.267656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:40.267295    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 11 14:09:41.267561 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:41.267525    2561 patch_prober.go:28] interesting pod/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 11 14:09:41.268003 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:41.267587    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 11 14:09:42.267617 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:42.267585    2561 patch_prober.go:28] interesting pod/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 11 14:09:42.268000 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:42.267643    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 11 14:09:43.267311 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:43.267269    2561 patch_prober.go:28] interesting pod/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 11 14:09:43.267501 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:43.267366    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 11 14:09:44.266974 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:44.266934    2561 patch_prober.go:28] interesting pod/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 11 14:09:44.267374 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:44.266990    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 11 14:09:45.004980 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:45.004944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:45.271159 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:45.271084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:45.271925 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:45.271893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:09:47.166242 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:09:47.166211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:10:03.095277 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:03.095240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:10:19.595109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:19.595079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:10:22.783118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:22.783083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"]
Sep 11 14:10:22.783532 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:22.783315    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6" podUID="c31f3169-bd75-4e3b-9b73-900e1be15c50" containerName="payload-processing" containerID="cri-o://6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f" gracePeriod=30
Sep 11 14:10:23.022285 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.022253    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:10:23.140815 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.140784    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f3169-bd75-4e3b-9b73-900e1be15c50-plugins-config-volume\") pod \"c31f3169-bd75-4e3b-9b73-900e1be15c50\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") "
Sep 11 14:10:23.141048 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.140838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtzwm\" (UniqueName: \"kubernetes.io/projected/c31f3169-bd75-4e3b-9b73-900e1be15c50-kube-api-access-gtzwm\") pod \"c31f3169-bd75-4e3b-9b73-900e1be15c50\" (UID: \"c31f3169-bd75-4e3b-9b73-900e1be15c50\") "
Sep 11 14:10:23.141249 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.141226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31f3169-bd75-4e3b-9b73-900e1be15c50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c31f3169-bd75-4e3b-9b73-900e1be15c50" (UID: "c31f3169-bd75-4e3b-9b73-900e1be15c50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:10:23.142831 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.142799    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31f3169-bd75-4e3b-9b73-900e1be15c50-kube-api-access-gtzwm" (OuterVolumeSpecName: "kube-api-access-gtzwm") pod "c31f3169-bd75-4e3b-9b73-900e1be15c50" (UID: "c31f3169-bd75-4e3b-9b73-900e1be15c50"). InnerVolumeSpecName "kube-api-access-gtzwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:10:23.242166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.242135    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f3169-bd75-4e3b-9b73-900e1be15c50-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:23.242166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.242165    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzwm\" (UniqueName: \"kubernetes.io/projected/c31f3169-bd75-4e3b-9b73-900e1be15c50-kube-api-access-gtzwm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:23.333806 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.333770    2561 generic.go:356] "Generic (PLEG): container finished" podID="c31f3169-bd75-4e3b-9b73-900e1be15c50" containerID="6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f" exitCode=0
Sep 11 14:10:23.334008 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.333820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"
Sep 11 14:10:23.334008 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.333860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6" event={"ID":"c31f3169-bd75-4e3b-9b73-900e1be15c50","Type":"ContainerDied","Data":"6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f"}
Sep 11 14:10:23.334008 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.333943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6" event={"ID":"c31f3169-bd75-4e3b-9b73-900e1be15c50","Type":"ContainerDied","Data":"7dfa4d9f5c3b7f5f68597a3f92434ded901ed6717114127f0bfdadf4365ac52a"}
Sep 11 14:10:23.334008 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.333976    2561 scope.go:117] "RemoveContainer" containerID="6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f"
Sep 11 14:10:23.342765 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.342744    2561 scope.go:117] "RemoveContainer" containerID="6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f"
Sep 11 14:10:23.343157 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:10:23.343138    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f\": container with ID starting with 6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f not found: ID does not exist" containerID="6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f"
Sep 11 14:10:23.343230 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.343170    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f"} err="failed to get container status \"6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f\": rpc error: code = NotFound desc = could not find container \"6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f\": container with ID starting with 6d8852d0d631700a673e4e2020096596c2ddcd6eb771ec99ab96d0812724642f not found: ID does not exist"
Sep 11 14:10:23.359810 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.359784    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"]
Sep 11 14:10:23.368073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:23.368051    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-25f5d3-6d99c7-9gjh6"]
Sep 11 14:10:24.117237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:24.117205    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31f3169-bd75-4e3b-9b73-900e1be15c50" path="/var/lib/kubelet/pods/c31f3169-bd75-4e3b-9b73-900e1be15c50/volumes"
Sep 11 14:10:30.489723 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:30.489693    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"]
Sep 11 14:10:30.490178 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:30.489896    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy" containerID="cri-o://5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa" gracePeriod=30
Sep 11 14:10:33.990475 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:33.990437    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:10:35.731866 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.731842    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:10:35.847932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.847885    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-token\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848127 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.847969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-certs\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848127 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848070    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-data\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848127 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848111    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-socket\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848160    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-podinfo\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848179    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-envoy\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-credential-socket\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848251    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istiod-ca-cert\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:10:35.848296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6lkh\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-kube-api-access-r6lkh\") pod \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\" (UID: \"2abd9a11-e015-48f6-a4f5-6657a03e2c6a\") "
Sep 11 14:10:35.848568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848318    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:10:35.848568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:10:35.848568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848543    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:10:35.848689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848626    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.848689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848643    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-data\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.848689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848657    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-workload-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.848689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848669    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-credential-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.848689 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.848665    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:10:35.850137 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.850103    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:10:35.850500 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.850480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:10:35.850663 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.850638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-kube-api-access-r6lkh" (OuterVolumeSpecName: "kube-api-access-r6lkh") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "kube-api-access-r6lkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:10:35.850756 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.850650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2abd9a11-e015-48f6-a4f5-6657a03e2c6a" (UID: "2abd9a11-e015-48f6-a4f5-6657a03e2c6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:10:35.949223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.949194    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-podinfo\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.949223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.949219    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-envoy\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.949223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.949230    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istiod-ca-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.949223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.949238    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lkh\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-kube-api-access-r6lkh\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:35.949479 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:35.949247    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abd9a11-e015-48f6-a4f5-6657a03e2c6a-istio-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:36.388989 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.388873    2561 generic.go:356] "Generic (PLEG): container finished" podID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerID="5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa" exitCode=0
Sep 11 14:10:36.388989 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.388953    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"
Sep 11 14:10:36.388989 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.388968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" event={"ID":"2abd9a11-e015-48f6-a4f5-6657a03e2c6a","Type":"ContainerDied","Data":"5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa"}
Sep 11 14:10:36.389223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.389052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd" event={"ID":"2abd9a11-e015-48f6-a4f5-6657a03e2c6a","Type":"ContainerDied","Data":"c17a367be01bcd4d34f32c125100d7064c79128063c5b5448108169bb51cc1b2"}
Sep 11 14:10:36.389223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.389077    2561 scope.go:117] "RemoveContainer" containerID="5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa"
Sep 11 14:10:36.397185 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.397159    2561 scope.go:117] "RemoveContainer" containerID="de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"
Sep 11 14:10:36.404389 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.404371    2561 scope.go:117] "RemoveContainer" containerID="5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa"
Sep 11 14:10:36.404642 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:10:36.404624    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa\": container with ID starting with 5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa not found: ID does not exist" containerID="5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa"
Sep 11 14:10:36.404730 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.404654    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa"} err="failed to get container status \"5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa\": rpc error: code = NotFound desc = could not find container \"5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa\": container with ID starting with 5b8b8e51f8fd84a5addf2436f5fb849486a6e2967fb674a58db199fd43ca99fa not found: ID does not exist"
Sep 11 14:10:36.404730 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.404676    2561 scope.go:117] "RemoveContainer" containerID="de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"
Sep 11 14:10:36.404901 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:10:36.404886    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a\": container with ID starting with de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a not found: ID does not exist" containerID="de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"
Sep 11 14:10:36.404976 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.404922    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a"} err="failed to get container status \"de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a\": rpc error: code = NotFound desc = could not find container \"de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a\": container with ID starting with de2074721e5b2f28e90370d9c3bdfdee0a099e49fc752d593aaf48b60174ea9a not found: ID does not exist"
Sep 11 14:10:36.415222 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.415201    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"]
Sep 11 14:10:36.422137 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:36.422115    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-25f5d3-openshift-default-5f6554c5d8-qw2kd"]
Sep 11 14:10:38.116267 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:38.116234    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" path="/var/lib/kubelet/pods/2abd9a11-e015-48f6-a4f5-6657a03e2c6a/volumes"
Sep 11 14:10:47.062337 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062300    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"]
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31f3169-bd75-4e3b-9b73-900e1be15c50" containerName="payload-processing"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31f3169-bd75-4e3b-9b73-900e1be15c50" containerName="payload-processing"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062594    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062600    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062609    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062614    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062669    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31f3169-bd75-4e3b-9b73-900e1be15c50" containerName="payload-processing"
Sep 11 14:10:47.062784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.062677    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy"
Sep 11 14:10:47.066573 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.066555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.074888 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.074865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f83766\"/\"kube-root-ca.crt\""
Sep 11 14:10:47.075113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.075096    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f83766\"/\"noten-model-f83766-kserve-self-signed-certs\""
Sep 11 14:10:47.075171 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.075111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f83766\"/\"openshift-service-ca.crt\""
Sep 11 14:10:47.075219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.075111    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f83766\"/\"default-dockercfg-2kjk6\""
Sep 11 14:10:47.157278 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.157245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"]
Sep 11 14:10:47.239305 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.239265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2zl\" (UniqueName: \"kubernetes.io/projected/d824fd73-8ad9-4677-b062-c6385b1c9d88-kube-api-access-nl2zl\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.239483 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.239315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-dshm\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.239483 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.239385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-home\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.239483 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.239447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-model-cache\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.239609 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.239485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824fd73-8ad9-4677-b062-c6385b1c9d88-tls-certs\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.239609 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.239507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-tmp-dir\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340378 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2zl\" (UniqueName: \"kubernetes.io/projected/d824fd73-8ad9-4677-b062-c6385b1c9d88-kube-api-access-nl2zl\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340378 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-dshm\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-home\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-model-cache\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340642 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824fd73-8ad9-4677-b062-c6385b1c9d88-tls-certs\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340642 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-tmp-dir\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.340987 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-home\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.341082 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.340993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-model-cache\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.341082 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.341010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-tmp-dir\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.342820 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.342795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-dshm\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.343192 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.343175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824fd73-8ad9-4677-b062-c6385b1c9d88-tls-certs\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.371739 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.371707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2zl\" (UniqueName: \"kubernetes.io/projected/d824fd73-8ad9-4677-b062-c6385b1c9d88-kube-api-access-nl2zl\") pod \"noten-model-f83766-kserve-548c9bd8bc-mkspk\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") " pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.375609 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.375586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:47.682366 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:47.682284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"]
Sep 11 14:10:47.693400 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:10:47.693355    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd824fd73_8ad9_4677_b062_c6385b1c9d88.slice/crio-732b1a316c03fbde6c7f154602b19b27c0342ac56a6f98caccd4092acc24f17b WatchSource:0}: Error finding container 732b1a316c03fbde6c7f154602b19b27c0342ac56a6f98caccd4092acc24f17b: Status 404 returned error can't find the container with id 732b1a316c03fbde6c7f154602b19b27c0342ac56a6f98caccd4092acc24f17b
Sep 11 14:10:48.447674 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:48.447628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk" event={"ID":"d824fd73-8ad9-4677-b062-c6385b1c9d88","Type":"ContainerStarted","Data":"29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5"}
Sep 11 14:10:48.448168 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:48.447684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk" event={"ID":"d824fd73-8ad9-4677-b062-c6385b1c9d88","Type":"ContainerStarted","Data":"732b1a316c03fbde6c7f154602b19b27c0342ac56a6f98caccd4092acc24f17b"}
Sep 11 14:10:48.504271 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:48.504214    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk" podStartSLOduration=2.504195857 podStartE2EDuration="2.504195857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:48.496031086 +0000 UTC m=+1096.863620530" watchObservedRunningTime="2026-09-11 14:10:48.504195857 +0000 UTC m=+1096.871785301"
Sep 11 14:10:48.865691 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:48.865660    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"]
Sep 11 14:10:50.613503 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:50.613468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:10:55.769011 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.768967    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"]
Sep 11 14:10:55.885163 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.880522    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"]
Sep 11 14:10:55.885163 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.881347    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abd9a11-e015-48f6-a4f5-6657a03e2c6a" containerName="istio-proxy"
Sep 11 14:10:55.885163 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.885003    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:55.889197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.889174    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:10:55.889351 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.889330    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:10:55.894569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.894545    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"]
Sep 11 14:10:55.909202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:55.909176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0587740-957c-4199-b864-348bc35ca6b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk\" (UID: \"d0587740-957c-4199-b864-348bc35ca6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:56.010270 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:56.010236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0587740-957c-4199-b864-348bc35ca6b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk\" (UID: \"d0587740-957c-4199-b864-348bc35ca6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:56.010829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:56.010811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0587740-957c-4199-b864-348bc35ca6b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk\" (UID: \"d0587740-957c-4199-b864-348bc35ca6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:56.197146 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:56.197113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:56.487256 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:56.487227    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"]
Sep 11 14:10:56.487480 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:10:56.487455    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0587740_957c_4199_b864_348bc35ca6b1.slice/crio-a03d0187ce58336038b171b76e5888d506214f254414688b7be3a7c180d83269 WatchSource:0}: Error finding container a03d0187ce58336038b171b76e5888d506214f254414688b7be3a7c180d83269: Status 404 returned error can't find the container with id a03d0187ce58336038b171b76e5888d506214f254414688b7be3a7c180d83269
Sep 11 14:10:56.862661 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:56.862625    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"]
Sep 11 14:10:57.376066 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:57.376037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:10:57.490140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:57.490052    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0587740-957c-4199-b864-348bc35ca6b1" containerID="b8a03de7c5e2237139a875bf001c7719fb1926e675b4ebc7c6619d68ee849402" exitCode=0
Sep 11 14:10:57.490140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:57.490100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk" event={"ID":"d0587740-957c-4199-b864-348bc35ca6b1","Type":"ContainerDied","Data":"b8a03de7c5e2237139a875bf001c7719fb1926e675b4ebc7c6619d68ee849402"}
Sep 11 14:10:57.490322 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:57.490163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk" event={"ID":"d0587740-957c-4199-b864-348bc35ca6b1","Type":"ContainerStarted","Data":"a03d0187ce58336038b171b76e5888d506214f254414688b7be3a7c180d83269"}
Sep 11 14:10:58.621547 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:58.621522    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:58.629487 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:58.629462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0587740-957c-4199-b864-348bc35ca6b1-maas-api-service-ca\") pod \"d0587740-957c-4199-b864-348bc35ca6b1\" (UID: \"d0587740-957c-4199-b864-348bc35ca6b1\") "
Sep 11 14:10:58.629796 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:58.629773    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0587740-957c-4199-b864-348bc35ca6b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0587740-957c-4199-b864-348bc35ca6b1" (UID: "d0587740-957c-4199-b864-348bc35ca6b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:10:58.730024 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:58.729986    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0587740-957c-4199-b864-348bc35ca6b1-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:10:59.499342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:59.499306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk" event={"ID":"d0587740-957c-4199-b864-348bc35ca6b1","Type":"ContainerDied","Data":"a03d0187ce58336038b171b76e5888d506214f254414688b7be3a7c180d83269"}
Sep 11 14:10:59.499342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:59.499339    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"
Sep 11 14:10:59.499614 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:10:59.499361    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a03d0187ce58336038b171b76e5888d506214f254414688b7be3a7c180d83269"
Sep 11 14:11:03.883994 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:03.883947    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" podUID="5b52c67b-2d36-4060-a768-f483ad246ab2" containerName="main" containerID="cri-o://cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1" gracePeriod=45
Sep 11 14:11:03.896645 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:03.896611    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" podUID="5b52c67b-2d36-4060-a768-f483ad246ab2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 11 14:11:04.141417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.141364    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:11:04.167003 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.166980    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-model-cache\") pod \"5b52c67b-2d36-4060-a768-f483ad246ab2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") "
Sep 11 14:11:04.167107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167020    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6wtz\" (UniqueName: \"kubernetes.io/projected/5b52c67b-2d36-4060-a768-f483ad246ab2-kube-api-access-m6wtz\") pod \"5b52c67b-2d36-4060-a768-f483ad246ab2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") "
Sep 11 14:11:04.167107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b52c67b-2d36-4060-a768-f483ad246ab2-tls-certs\") pod \"5b52c67b-2d36-4060-a768-f483ad246ab2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") "
Sep 11 14:11:04.167107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-tmp-dir\") pod \"5b52c67b-2d36-4060-a768-f483ad246ab2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") "
Sep 11 14:11:04.167220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167167    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-home\") pod \"5b52c67b-2d36-4060-a768-f483ad246ab2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") "
Sep 11 14:11:04.167220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-dshm\") pod \"5b52c67b-2d36-4060-a768-f483ad246ab2\" (UID: \"5b52c67b-2d36-4060-a768-f483ad246ab2\") "
Sep 11 14:11:04.167322 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b52c67b-2d36-4060-a768-f483ad246ab2" (UID: "5b52c67b-2d36-4060-a768-f483ad246ab2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:04.167474 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167406    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-home" (OuterVolumeSpecName: "home") pod "5b52c67b-2d36-4060-a768-f483ad246ab2" (UID: "5b52c67b-2d36-4060-a768-f483ad246ab2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:04.167474 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167426    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b52c67b-2d36-4060-a768-f483ad246ab2" (UID: "5b52c67b-2d36-4060-a768-f483ad246ab2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:04.167645 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167483    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:04.167645 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167498    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:04.167645 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.167513    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:04.169396 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.169372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-dshm" (OuterVolumeSpecName: "dshm") pod "5b52c67b-2d36-4060-a768-f483ad246ab2" (UID: "5b52c67b-2d36-4060-a768-f483ad246ab2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:04.169478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.169391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b52c67b-2d36-4060-a768-f483ad246ab2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b52c67b-2d36-4060-a768-f483ad246ab2" (UID: "5b52c67b-2d36-4060-a768-f483ad246ab2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:11:04.169478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.169433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b52c67b-2d36-4060-a768-f483ad246ab2-kube-api-access-m6wtz" (OuterVolumeSpecName: "kube-api-access-m6wtz") pod "5b52c67b-2d36-4060-a768-f483ad246ab2" (UID: "5b52c67b-2d36-4060-a768-f483ad246ab2"). InnerVolumeSpecName "kube-api-access-m6wtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:04.267927 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.267886    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b52c67b-2d36-4060-a768-f483ad246ab2-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:04.268083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.267934    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6wtz\" (UniqueName: \"kubernetes.io/projected/5b52c67b-2d36-4060-a768-f483ad246ab2-kube-api-access-m6wtz\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:04.268083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.267946    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b52c67b-2d36-4060-a768-f483ad246ab2-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:04.521631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.521548    2561 generic.go:356] "Generic (PLEG): container finished" podID="5b52c67b-2d36-4060-a768-f483ad246ab2" containerID="cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1" exitCode=0
Sep 11 14:11:04.521631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.521597    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"
Sep 11 14:11:04.521821 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.521630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" event={"ID":"5b52c67b-2d36-4060-a768-f483ad246ab2","Type":"ContainerDied","Data":"cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1"}
Sep 11 14:11:04.521821 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.521670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2" event={"ID":"5b52c67b-2d36-4060-a768-f483ad246ab2","Type":"ContainerDied","Data":"619e7b31551ad24877b66a3c906681d729d06e0fe17d76038b9b28afb7cb40ce"}
Sep 11 14:11:04.521821 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.521686    2561 scope.go:117] "RemoveContainer" containerID="cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1"
Sep 11 14:11:04.530163 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.530147    2561 scope.go:117] "RemoveContainer" containerID="cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1"
Sep 11 14:11:04.530430 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:04.530412    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1\": container with ID starting with cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1 not found: ID does not exist" containerID="cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1"
Sep 11 14:11:04.530482 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.530440    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1"} err="failed to get container status \"cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1\": rpc error: code = NotFound desc = could not find container \"cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1\": container with ID starting with cecc80cb1a2b1f02b3e8265d07707cce137adf9d18eecb9061a4b5461dea75e1 not found: ID does not exist"
Sep 11 14:11:04.549020 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.548997    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"]
Sep 11 14:11:04.556023 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:04.556004    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-abfca2/test-model-abfca2-kserve-75c54d8d55-wgrb2"]
Sep 11 14:11:06.116979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:06.116948    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b52c67b-2d36-4060-a768-f483ad246ab2" path="/var/lib/kubelet/pods/5b52c67b-2d36-4060-a768-f483ad246ab2/volumes"
Sep 11 14:11:06.196372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:06.196334    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:11:09.853205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:09.853165    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"]
Sep 11 14:11:09.853667 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:09.853425    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh" podUID="54feb8e7-d36f-47d3-97ea-c053f86909d9" containerName="payload-pre-processing" containerID="cri-o://c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98" gracePeriod=30
Sep 11 14:11:10.109218 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.109194    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:11:10.220316 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.220284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54feb8e7-d36f-47d3-97ea-c053f86909d9-plugins-config-volume\") pod \"54feb8e7-d36f-47d3-97ea-c053f86909d9\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") "
Sep 11 14:11:10.220511 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.220324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx5cq\" (UniqueName: \"kubernetes.io/projected/54feb8e7-d36f-47d3-97ea-c053f86909d9-kube-api-access-kx5cq\") pod \"54feb8e7-d36f-47d3-97ea-c053f86909d9\" (UID: \"54feb8e7-d36f-47d3-97ea-c053f86909d9\") "
Sep 11 14:11:10.220659 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.220633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54feb8e7-d36f-47d3-97ea-c053f86909d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54feb8e7-d36f-47d3-97ea-c053f86909d9" (UID: "54feb8e7-d36f-47d3-97ea-c053f86909d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:11:10.222470 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.222447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54feb8e7-d36f-47d3-97ea-c053f86909d9-kube-api-access-kx5cq" (OuterVolumeSpecName: "kube-api-access-kx5cq") pod "54feb8e7-d36f-47d3-97ea-c053f86909d9" (UID: "54feb8e7-d36f-47d3-97ea-c053f86909d9"). InnerVolumeSpecName "kube-api-access-kx5cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:10.321603 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.321571    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54feb8e7-d36f-47d3-97ea-c053f86909d9-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:10.321603 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.321598    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5cq\" (UniqueName: \"kubernetes.io/projected/54feb8e7-d36f-47d3-97ea-c053f86909d9-kube-api-access-kx5cq\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:10.554097 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.554064    2561 generic.go:356] "Generic (PLEG): container finished" podID="54feb8e7-d36f-47d3-97ea-c053f86909d9" containerID="c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98" exitCode=0
Sep 11 14:11:10.554292 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.554119    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"
Sep 11 14:11:10.554292 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.554114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh" event={"ID":"54feb8e7-d36f-47d3-97ea-c053f86909d9","Type":"ContainerDied","Data":"c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98"}
Sep 11 14:11:10.554292 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.554243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh" event={"ID":"54feb8e7-d36f-47d3-97ea-c053f86909d9","Type":"ContainerDied","Data":"76676a7eb3fdedec6e4ffecedec91c0656813ca07bcf16bcaf2e7e3e9dc810b0"}
Sep 11 14:11:10.554292 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.554271    2561 scope.go:117] "RemoveContainer" containerID="c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98"
Sep 11 14:11:10.563034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.563015    2561 scope.go:117] "RemoveContainer" containerID="c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98"
Sep 11 14:11:10.563336 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:10.563316    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98\": container with ID starting with c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98 not found: ID does not exist" containerID="c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98"
Sep 11 14:11:10.563394 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.563345    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98"} err="failed to get container status \"c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98\": rpc error: code = NotFound desc = could not find container \"c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98\": container with ID starting with c7253ec3bdab49eace0717772111951fa3561eca4c06bb96d131080bbab37d98 not found: ID does not exist"
Sep 11 14:11:10.567667 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.567646    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"]
Sep 11 14:11:10.576624 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.576566    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-abfca2-zxtrk"]
Sep 11 14:11:10.618347 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.618319    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"]
Sep 11 14:11:10.642477 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.642448    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-abfca2-5d488f867d-49gsh"]
Sep 11 14:11:10.787537 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:10.787475    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t" podUID="3d581bf4-c624-459a-a680-e44fc4127df8" containerName="main" containerID="cri-o://48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc" gracePeriod=45
Sep 11 14:11:11.034430 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.034406    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:11:11.128937 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.128892    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d581bf4-c624-459a-a680-e44fc4127df8-tls-certs\") pod \"3d581bf4-c624-459a-a680-e44fc4127df8\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") "
Sep 11 14:11:11.129126 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.128988    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6znf\" (UniqueName: \"kubernetes.io/projected/3d581bf4-c624-459a-a680-e44fc4127df8-kube-api-access-g6znf\") pod \"3d581bf4-c624-459a-a680-e44fc4127df8\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") "
Sep 11 14:11:11.129126 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-tmp-dir\") pod \"3d581bf4-c624-459a-a680-e44fc4127df8\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") "
Sep 11 14:11:11.129126 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-home\") pod \"3d581bf4-c624-459a-a680-e44fc4127df8\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") "
Sep 11 14:11:11.129126 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-model-cache\") pod \"3d581bf4-c624-459a-a680-e44fc4127df8\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") "
Sep 11 14:11:11.129126 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-dshm\") pod \"3d581bf4-c624-459a-a680-e44fc4127df8\" (UID: \"3d581bf4-c624-459a-a680-e44fc4127df8\") "
Sep 11 14:11:11.129412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-home" (OuterVolumeSpecName: "home") pod "3d581bf4-c624-459a-a680-e44fc4127df8" (UID: "3d581bf4-c624-459a-a680-e44fc4127df8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:11.129412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d581bf4-c624-459a-a680-e44fc4127df8" (UID: "3d581bf4-c624-459a-a680-e44fc4127df8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:11.129412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.129351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d581bf4-c624-459a-a680-e44fc4127df8" (UID: "3d581bf4-c624-459a-a680-e44fc4127df8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:11.131197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.131166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d581bf4-c624-459a-a680-e44fc4127df8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d581bf4-c624-459a-a680-e44fc4127df8" (UID: "3d581bf4-c624-459a-a680-e44fc4127df8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:11:11.131197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.131182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d581bf4-c624-459a-a680-e44fc4127df8-kube-api-access-g6znf" (OuterVolumeSpecName: "kube-api-access-g6znf") pod "3d581bf4-c624-459a-a680-e44fc4127df8" (UID: "3d581bf4-c624-459a-a680-e44fc4127df8"). InnerVolumeSpecName "kube-api-access-g6znf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:11.131362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.131335    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-dshm" (OuterVolumeSpecName: "dshm") pod "3d581bf4-c624-459a-a680-e44fc4127df8" (UID: "3d581bf4-c624-459a-a680-e44fc4127df8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:11.230034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.229950    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6znf\" (UniqueName: \"kubernetes.io/projected/3d581bf4-c624-459a-a680-e44fc4127df8-kube-api-access-g6znf\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:11.230034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.229979    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:11.230034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.229991    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:11.230034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.229999    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:11.230034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.230006    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d581bf4-c624-459a-a680-e44fc4127df8-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:11.230034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.230014    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d581bf4-c624-459a-a680-e44fc4127df8-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:11.560260 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.560229    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d581bf4-c624-459a-a680-e44fc4127df8" containerID="48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc" exitCode=0
Sep 11 14:11:11.560436 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.560270    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"
Sep 11 14:11:11.560436 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.560321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t" event={"ID":"3d581bf4-c624-459a-a680-e44fc4127df8","Type":"ContainerDied","Data":"48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc"}
Sep 11 14:11:11.560436 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.560362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t" event={"ID":"3d581bf4-c624-459a-a680-e44fc4127df8","Type":"ContainerDied","Data":"707f850c6fd67403931491afd0e014195cc0015ffa74d8e95efc62a0dd5f4b4f"}
Sep 11 14:11:11.560436 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.560381    2561 scope.go:117] "RemoveContainer" containerID="48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc"
Sep 11 14:11:11.569297 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.569281    2561 scope.go:117] "RemoveContainer" containerID="48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc"
Sep 11 14:11:11.569542 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:11.569523    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc\": container with ID starting with 48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc not found: ID does not exist" containerID="48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc"
Sep 11 14:11:11.569639 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.569551    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc"} err="failed to get container status \"48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc\": rpc error: code = NotFound desc = could not find container \"48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc\": container with ID starting with 48ca7bb9bcc9cdf25df7cd5cf7e22a18275b97876741036f66f1ab9635bf8bfc not found: ID does not exist"
Sep 11 14:11:11.609895 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.609862    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"]
Sep 11 14:11:11.620930 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.620882    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-80fb45/test-model-80fb45-kserve-77b44886bc-g6f2t"]
Sep 11 14:11:11.815776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.815688    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"]
Sep 11 14:11:11.815981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.815968    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b52c67b-2d36-4060-a768-f483ad246ab2" containerName="main"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.815983    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b52c67b-2d36-4060-a768-f483ad246ab2" containerName="main"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.815999    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54feb8e7-d36f-47d3-97ea-c053f86909d9" containerName="payload-pre-processing"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816005    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54feb8e7-d36f-47d3-97ea-c053f86909d9" containerName="payload-pre-processing"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816014    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d581bf4-c624-459a-a680-e44fc4127df8" containerName="main"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816019    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d581bf4-c624-459a-a680-e44fc4127df8" containerName="main"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816030    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0587740-957c-4199-b864-348bc35ca6b1" containerName="revoke-keys"
Sep 11 14:11:11.816039 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816035    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0587740-957c-4199-b864-348bc35ca6b1" containerName="revoke-keys"
Sep 11 14:11:11.816241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816100    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d581bf4-c624-459a-a680-e44fc4127df8" containerName="main"
Sep 11 14:11:11.816241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816107    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54feb8e7-d36f-47d3-97ea-c053f86909d9" containerName="payload-pre-processing"
Sep 11 14:11:11.816241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816115    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b52c67b-2d36-4060-a768-f483ad246ab2" containerName="main"
Sep 11 14:11:11.816241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.816123    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0587740-957c-4199-b864-348bc35ca6b1" containerName="revoke-keys"
Sep 11 14:11:11.820173 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.820156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:11.833114 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.833092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3e88320d\""
Sep 11 14:11:11.841936 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.841894    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"]
Sep 11 14:11:11.854525 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.854401    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"]
Sep 11 14:11:11.858041 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.858016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:11.872829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.872804    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"]
Sep 11 14:11:11.881347 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.881312    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk" podUID="d824fd73-8ad9-4677-b062-c6385b1c9d88" containerName="main" containerID="cri-o://29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5" gracePeriod=45
Sep 11 14:11:11.933369 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.933344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bnnz\" (UniqueName: \"kubernetes.io/projected/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-kube-api-access-8bnnz\") pod \"payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") " pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:11.933475 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.933379    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe2a114-9c21-45e1-9323-001aee5845b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:11.933475 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.933431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-plugins-config-volume\") pod \"payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") " pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:11.933475 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:11.933461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szr7q\" (UniqueName: \"kubernetes.io/projected/abe2a114-9c21-45e1-9323-001aee5845b8-kube-api-access-szr7q\") pod \"payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.034004 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.033974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szr7q\" (UniqueName: \"kubernetes.io/projected/abe2a114-9c21-45e1-9323-001aee5845b8-kube-api-access-szr7q\") pod \"payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.034149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.034034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bnnz\" (UniqueName: \"kubernetes.io/projected/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-kube-api-access-8bnnz\") pod \"payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") " pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:12.034149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.034062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe2a114-9c21-45e1-9323-001aee5845b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.034149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.034098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-plugins-config-volume\") pod \"payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") " pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:12.034973 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.034949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe2a114-9c21-45e1-9323-001aee5845b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.035250 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.034949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-plugins-config-volume\") pod \"payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") " pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:12.046639 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.046595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szr7q\" (UniqueName: \"kubernetes.io/projected/abe2a114-9c21-45e1-9323-001aee5845b8-kube-api-access-szr7q\") pod \"payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.048143 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.048109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bnnz\" (UniqueName: \"kubernetes.io/projected/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-kube-api-access-8bnnz\") pod \"payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") " pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:12.116315 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.116282    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d581bf4-c624-459a-a680-e44fc4127df8" path="/var/lib/kubelet/pods/3d581bf4-c624-459a-a680-e44fc4127df8/volumes"
Sep 11 14:11:12.116777 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.116758    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54feb8e7-d36f-47d3-97ea-c053f86909d9" path="/var/lib/kubelet/pods/54feb8e7-d36f-47d3-97ea-c053f86909d9/volumes"
Sep 11 14:11:12.117224 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.117206    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0587740-957c-4199-b864-348bc35ca6b1" path="/var/lib/kubelet/pods/d0587740-957c-4199-b864-348bc35ca6b1/volumes"
Sep 11 14:11:12.123668 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.123649    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:11:12.129342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.129327    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:12.169771 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.169740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.236533 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-home\") pod \"d824fd73-8ad9-4677-b062-c6385b1c9d88\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") "
Sep 11 14:11:12.236662 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236580    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-model-cache\") pod \"d824fd73-8ad9-4677-b062-c6385b1c9d88\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") "
Sep 11 14:11:12.236662 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824fd73-8ad9-4677-b062-c6385b1c9d88-tls-certs\") pod \"d824fd73-8ad9-4677-b062-c6385b1c9d88\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") "
Sep 11 14:11:12.236784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl2zl\" (UniqueName: \"kubernetes.io/projected/d824fd73-8ad9-4677-b062-c6385b1c9d88-kube-api-access-nl2zl\") pod \"d824fd73-8ad9-4677-b062-c6385b1c9d88\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") "
Sep 11 14:11:12.236784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-dshm\") pod \"d824fd73-8ad9-4677-b062-c6385b1c9d88\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") "
Sep 11 14:11:12.236784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236753    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-tmp-dir\") pod \"d824fd73-8ad9-4677-b062-c6385b1c9d88\" (UID: \"d824fd73-8ad9-4677-b062-c6385b1c9d88\") "
Sep 11 14:11:12.236964 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.236802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-home" (OuterVolumeSpecName: "home") pod "d824fd73-8ad9-4677-b062-c6385b1c9d88" (UID: "d824fd73-8ad9-4677-b062-c6385b1c9d88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:12.237319 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.237018    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:12.237319 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.237261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-model-cache" (OuterVolumeSpecName: "model-cache") pod "d824fd73-8ad9-4677-b062-c6385b1c9d88" (UID: "d824fd73-8ad9-4677-b062-c6385b1c9d88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:12.237319 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.237275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d824fd73-8ad9-4677-b062-c6385b1c9d88" (UID: "d824fd73-8ad9-4677-b062-c6385b1c9d88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:12.240057 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.240033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-dshm" (OuterVolumeSpecName: "dshm") pod "d824fd73-8ad9-4677-b062-c6385b1c9d88" (UID: "d824fd73-8ad9-4677-b062-c6385b1c9d88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:11:12.240155 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.240080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d824fd73-8ad9-4677-b062-c6385b1c9d88-kube-api-access-nl2zl" (OuterVolumeSpecName: "kube-api-access-nl2zl") pod "d824fd73-8ad9-4677-b062-c6385b1c9d88" (UID: "d824fd73-8ad9-4677-b062-c6385b1c9d88"). InnerVolumeSpecName "kube-api-access-nl2zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:12.240155 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.240137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d824fd73-8ad9-4677-b062-c6385b1c9d88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d824fd73-8ad9-4677-b062-c6385b1c9d88" (UID: "d824fd73-8ad9-4677-b062-c6385b1c9d88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:11:12.337932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.337884    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:12.337932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.337926    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824fd73-8ad9-4677-b062-c6385b1c9d88-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:12.337932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.337937    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2zl\" (UniqueName: \"kubernetes.io/projected/d824fd73-8ad9-4677-b062-c6385b1c9d88-kube-api-access-nl2zl\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:12.338112 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.337946    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:12.338112 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.337954    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824fd73-8ad9-4677-b062-c6385b1c9d88-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:12.404140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.404117    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"]
Sep 11 14:11:12.406541 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:11:12.406517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d7de3d_e326_4d4b_b972_e81bfd23f2e0.slice/crio-b5f3aef58ebbae2cf2d800642115b1b318cd4bf69cf6d966fb422df2264312a5 WatchSource:0}: Error finding container b5f3aef58ebbae2cf2d800642115b1b318cd4bf69cf6d966fb422df2264312a5: Status 404 returned error can't find the container with id b5f3aef58ebbae2cf2d800642115b1b318cd4bf69cf6d966fb422df2264312a5
Sep 11 14:11:12.451932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.451887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"]
Sep 11 14:11:12.453665 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:11:12.453642    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe2a114_9c21_45e1_9323_001aee5845b8.slice/crio-9794eaeea6574d42af17d4412fb3cbb95339b334f74f16021685a48a064f8c6a WatchSource:0}: Error finding container 9794eaeea6574d42af17d4412fb3cbb95339b334f74f16021685a48a064f8c6a: Status 404 returned error can't find the container with id 9794eaeea6574d42af17d4412fb3cbb95339b334f74f16021685a48a064f8c6a
Sep 11 14:11:12.567568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.567539    2561 generic.go:356] "Generic (PLEG): container finished" podID="d824fd73-8ad9-4677-b062-c6385b1c9d88" containerID="29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5" exitCode=0
Sep 11 14:11:12.567724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.567595    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"
Sep 11 14:11:12.567724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.567613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk" event={"ID":"d824fd73-8ad9-4677-b062-c6385b1c9d88","Type":"ContainerDied","Data":"29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5"}
Sep 11 14:11:12.567724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.567652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk" event={"ID":"d824fd73-8ad9-4677-b062-c6385b1c9d88","Type":"ContainerDied","Data":"732b1a316c03fbde6c7f154602b19b27c0342ac56a6f98caccd4092acc24f17b"}
Sep 11 14:11:12.567724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.567678    2561 scope.go:117] "RemoveContainer" containerID="29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5"
Sep 11 14:11:12.570046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.569722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8" event={"ID":"abe2a114-9c21-45e1-9323-001aee5845b8","Type":"ContainerStarted","Data":"e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c"}
Sep 11 14:11:12.570046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.569772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8" event={"ID":"abe2a114-9c21-45e1-9323-001aee5845b8","Type":"ContainerStarted","Data":"9794eaeea6574d42af17d4412fb3cbb95339b334f74f16021685a48a064f8c6a"}
Sep 11 14:11:12.570046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.569824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:12.573095 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.573061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm" event={"ID":"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0","Type":"ContainerStarted","Data":"e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7"}
Sep 11 14:11:12.573221 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.573105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm" event={"ID":"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0","Type":"ContainerStarted","Data":"b5f3aef58ebbae2cf2d800642115b1b318cd4bf69cf6d966fb422df2264312a5"}
Sep 11 14:11:12.573221 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.573134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:12.578337 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.578311    2561 scope.go:117] "RemoveContainer" containerID="29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5"
Sep 11 14:11:12.578596 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:12.578577    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5\": container with ID starting with 29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5 not found: ID does not exist" containerID="29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5"
Sep 11 14:11:12.578665 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.578603    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5"} err="failed to get container status \"29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5\": rpc error: code = NotFound desc = could not find container \"29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5\": container with ID starting with 29f454f36bea57bc808bda7799cb305fc09b72d2028dcb46fcc636334ff417a5 not found: ID does not exist"
Sep 11 14:11:12.597588 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.597547    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8" podStartSLOduration=1.597534268 podStartE2EDuration="1.597534268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:12.595982574 +0000 UTC m=+1120.963572016" watchObservedRunningTime="2026-09-11 14:11:12.597534268 +0000 UTC m=+1120.965123711"
Sep 11 14:11:12.623811 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.623759    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm" podStartSLOduration=1.623742928 podStartE2EDuration="1.623742928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:12.622554449 +0000 UTC m=+1120.990143905" watchObservedRunningTime="2026-09-11 14:11:12.623742928 +0000 UTC m=+1120.991332374"
Sep 11 14:11:12.641410 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.641387    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"]
Sep 11 14:11:12.646967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:12.646935    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f83766/noten-model-f83766-kserve-548c9bd8bc-mkspk"]
Sep 11 14:11:14.116952 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:14.116903    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d824fd73-8ad9-4677-b062-c6385b1c9d88" path="/var/lib/kubelet/pods/d824fd73-8ad9-4677-b062-c6385b1c9d88/volumes"
Sep 11 14:11:17.706775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:17.706742    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:11:18.512249 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.512208    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"]
Sep 11 14:11:18.512589 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.512572    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d824fd73-8ad9-4677-b062-c6385b1c9d88" containerName="main"
Sep 11 14:11:18.512670 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.512592    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d824fd73-8ad9-4677-b062-c6385b1c9d88" containerName="main"
Sep 11 14:11:18.512725 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.512700    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d824fd73-8ad9-4677-b062-c6385b1c9d88" containerName="main"
Sep 11 14:11:18.517073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.517051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:18.520365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.520344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:11:18.520754 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.520739    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:11:18.534608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.534577    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"]
Sep 11 14:11:18.583841 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.583814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8924935-94ae-4902-9030-968193955c30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz\" (UID: \"d8924935-94ae-4902-9030-968193955c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:18.684520 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.684475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8924935-94ae-4902-9030-968193955c30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz\" (UID: \"d8924935-94ae-4902-9030-968193955c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:18.685131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.685113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8924935-94ae-4902-9030-968193955c30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz\" (UID: \"d8924935-94ae-4902-9030-968193955c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:18.827273 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:18.827246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:19.098001 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:19.097975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"]
Sep 11 14:11:19.100017 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:11:19.099992    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8924935_94ae_4902_9030_968193955c30.slice/crio-b157bcb4ebfad9dfa3afc9ac492c34de1a6be52b6a9dbfa23bb4554d176f53c8 WatchSource:0}: Error finding container b157bcb4ebfad9dfa3afc9ac492c34de1a6be52b6a9dbfa23bb4554d176f53c8: Status 404 returned error can't find the container with id b157bcb4ebfad9dfa3afc9ac492c34de1a6be52b6a9dbfa23bb4554d176f53c8
Sep 11 14:11:19.605868 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:19.605837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz" event={"ID":"d8924935-94ae-4902-9030-968193955c30","Type":"ContainerStarted","Data":"2c871bbb8729435e38b72f720351f459fdd8de24978f2891c1ff603cc18775c4"}
Sep 11 14:11:19.606077 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:19.605885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz" event={"ID":"d8924935-94ae-4902-9030-968193955c30","Type":"ContainerStarted","Data":"b157bcb4ebfad9dfa3afc9ac492c34de1a6be52b6a9dbfa23bb4554d176f53c8"}
Sep 11 14:11:19.628020 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:19.627973    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz" podStartSLOduration=1.627957893 podStartE2EDuration="1.627957893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:19.626656307 +0000 UTC m=+1127.994245750" watchObservedRunningTime="2026-09-11 14:11:19.627957893 +0000 UTC m=+1127.995547337"
Sep 11 14:11:20.611861 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:20.611825    2561 generic.go:356] "Generic (PLEG): container finished" podID="d8924935-94ae-4902-9030-968193955c30" containerID="2c871bbb8729435e38b72f720351f459fdd8de24978f2891c1ff603cc18775c4" exitCode=0
Sep 11 14:11:20.612279 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:20.611931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz" event={"ID":"d8924935-94ae-4902-9030-968193955c30","Type":"ContainerDied","Data":"2c871bbb8729435e38b72f720351f459fdd8de24978f2891c1ff603cc18775c4"}
Sep 11 14:11:21.743101 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:21.743077    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:21.807264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:21.807234    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8924935-94ae-4902-9030-968193955c30-maas-api-service-ca\") pod \"d8924935-94ae-4902-9030-968193955c30\" (UID: \"d8924935-94ae-4902-9030-968193955c30\") "
Sep 11 14:11:21.807566 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:21.807544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8924935-94ae-4902-9030-968193955c30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8924935-94ae-4902-9030-968193955c30" (UID: "d8924935-94ae-4902-9030-968193955c30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:11:21.908820 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:21.908746    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8924935-94ae-4902-9030-968193955c30-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:22.623489 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:22.623458    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"
Sep 11 14:11:22.623696 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:22.623457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz" event={"ID":"d8924935-94ae-4902-9030-968193955c30","Type":"ContainerDied","Data":"b157bcb4ebfad9dfa3afc9ac492c34de1a6be52b6a9dbfa23bb4554d176f53c8"}
Sep 11 14:11:22.623696 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:22.623598    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b157bcb4ebfad9dfa3afc9ac492c34de1a6be52b6a9dbfa23bb4554d176f53c8"
Sep 11 14:11:23.579645 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:23.579616    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:23.580061 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:23.579844    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:33.848175 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:33.848145    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"]
Sep 11 14:11:33.848547 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:33.848388    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn" podUID="745de085-2ec9-46c9-9f37-507dbd5e299e" containerName="payload-processing" containerID="cri-o://727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88" gracePeriod=30
Sep 11 14:11:34.099763 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.099701    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:11:34.177111 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.177080    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"]
Sep 11 14:11:34.180703 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.180679    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-80fb45-s95jz"]
Sep 11 14:11:34.208511 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.208482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb6xg\" (UniqueName: \"kubernetes.io/projected/745de085-2ec9-46c9-9f37-507dbd5e299e-kube-api-access-lb6xg\") pod \"745de085-2ec9-46c9-9f37-507dbd5e299e\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") "
Sep 11 14:11:34.208658 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.208528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745de085-2ec9-46c9-9f37-507dbd5e299e-plugins-config-volume\") pod \"745de085-2ec9-46c9-9f37-507dbd5e299e\" (UID: \"745de085-2ec9-46c9-9f37-507dbd5e299e\") "
Sep 11 14:11:34.209054 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.209014    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745de085-2ec9-46c9-9f37-507dbd5e299e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "745de085-2ec9-46c9-9f37-507dbd5e299e" (UID: "745de085-2ec9-46c9-9f37-507dbd5e299e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:11:34.210726 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.210701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745de085-2ec9-46c9-9f37-507dbd5e299e-kube-api-access-lb6xg" (OuterVolumeSpecName: "kube-api-access-lb6xg") pod "745de085-2ec9-46c9-9f37-507dbd5e299e" (UID: "745de085-2ec9-46c9-9f37-507dbd5e299e"). InnerVolumeSpecName "kube-api-access-lb6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:34.310009 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.309975    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb6xg\" (UniqueName: \"kubernetes.io/projected/745de085-2ec9-46c9-9f37-507dbd5e299e-kube-api-access-lb6xg\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:34.310009 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.310009    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745de085-2ec9-46c9-9f37-507dbd5e299e-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:34.675149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.675109    2561 generic.go:356] "Generic (PLEG): container finished" podID="745de085-2ec9-46c9-9f37-507dbd5e299e" containerID="727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88" exitCode=0
Sep 11 14:11:34.675330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.675172    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"
Sep 11 14:11:34.675330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.675192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn" event={"ID":"745de085-2ec9-46c9-9f37-507dbd5e299e","Type":"ContainerDied","Data":"727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88"}
Sep 11 14:11:34.675330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.675268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn" event={"ID":"745de085-2ec9-46c9-9f37-507dbd5e299e","Type":"ContainerDied","Data":"072b80dc7e2cedf01b9ed511008e6ccf8005485eec619c1305a307cd21c801af"}
Sep 11 14:11:34.675330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.675290    2561 scope.go:117] "RemoveContainer" containerID="727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88"
Sep 11 14:11:34.684122 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.684105    2561 scope.go:117] "RemoveContainer" containerID="727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88"
Sep 11 14:11:34.684382 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:34.684365    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88\": container with ID starting with 727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88 not found: ID does not exist" containerID="727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88"
Sep 11 14:11:34.684437 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.684390    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88"} err="failed to get container status \"727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88\": rpc error: code = NotFound desc = could not find container \"727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88\": container with ID starting with 727be1717ea81982049426e502212e0865cb48bfea8be542646181c4b93fba88 not found: ID does not exist"
Sep 11 14:11:34.696643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.696619    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"]
Sep 11 14:11:34.702885 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:34.702862    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-80fb45-787b4f5c6f-bjqhn"]
Sep 11 14:11:36.118266 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.118232    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745de085-2ec9-46c9-9f37-507dbd5e299e" path="/var/lib/kubelet/pods/745de085-2ec9-46c9-9f37-507dbd5e299e/volumes"
Sep 11 14:11:36.118635 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.118528    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8924935-94ae-4902-9030-968193955c30" path="/var/lib/kubelet/pods/d8924935-94ae-4902-9030-968193955c30/volumes"
Sep 11 14:11:36.625327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625295    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"]
Sep 11 14:11:36.625587 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745de085-2ec9-46c9-9f37-507dbd5e299e" containerName="payload-processing"
Sep 11 14:11:36.625631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625589    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="745de085-2ec9-46c9-9f37-507dbd5e299e" containerName="payload-processing"
Sep 11 14:11:36.625631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8924935-94ae-4902-9030-968193955c30" containerName="revoke-keys"
Sep 11 14:11:36.625631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625615    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8924935-94ae-4902-9030-968193955c30" containerName="revoke-keys"
Sep 11 14:11:36.625723 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625668    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="745de085-2ec9-46c9-9f37-507dbd5e299e" containerName="payload-processing"
Sep 11 14:11:36.625723 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.625683    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8924935-94ae-4902-9030-968193955c30" containerName="revoke-keys"
Sep 11 14:11:36.630022 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.630005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:36.633481 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.633461    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:11:36.633756 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.633740    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:11:36.650394 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.650360    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"]
Sep 11 14:11:36.726372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.726340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3e88320d-89tkp\" (UID: \"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:36.769417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.769386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:11:36.827075 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.827045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3e88320d-89tkp\" (UID: \"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:36.827739 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.827716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3e88320d-89tkp\" (UID: \"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:36.940410 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:36.940335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:37.210900 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:37.210828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"]
Sep 11 14:11:37.214029 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:11:37.214001    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbf1b1b_bf18_44f3_ad9c_d461e1869baf.slice/crio-f99fb2359f85ee8f04b15e861280ef19e809d8f3ad00c7dc5814ddd22215520c WatchSource:0}: Error finding container f99fb2359f85ee8f04b15e861280ef19e809d8f3ad00c7dc5814ddd22215520c: Status 404 returned error can't find the container with id f99fb2359f85ee8f04b15e861280ef19e809d8f3ad00c7dc5814ddd22215520c
Sep 11 14:11:37.689732 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:37.689696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp" event={"ID":"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf","Type":"ContainerStarted","Data":"ce8afa3e6a522733d3614880e894727b72d597afa702021ec2d0f1ead2cb88b4"}
Sep 11 14:11:37.689890 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:37.689755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp" event={"ID":"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf","Type":"ContainerStarted","Data":"f99fb2359f85ee8f04b15e861280ef19e809d8f3ad00c7dc5814ddd22215520c"}
Sep 11 14:11:37.743645 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:37.743593    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp" podStartSLOduration=1.743576675 podStartE2EDuration="1.743576675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:37.742043548 +0000 UTC m=+1146.109632991" watchObservedRunningTime="2026-09-11 14:11:37.743576675 +0000 UTC m=+1146.111166117"
Sep 11 14:11:38.694883 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:38.694850    2561 generic.go:356] "Generic (PLEG): container finished" podID="cbbf1b1b-bf18-44f3-ad9c-d461e1869baf" containerID="ce8afa3e6a522733d3614880e894727b72d597afa702021ec2d0f1ead2cb88b4" exitCode=0
Sep 11 14:11:38.695290 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:38.694940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp" event={"ID":"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf","Type":"ContainerDied","Data":"ce8afa3e6a522733d3614880e894727b72d597afa702021ec2d0f1ead2cb88b4"}
Sep 11 14:11:39.823967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:39.823944    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:39.953355 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:39.953278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf-maas-api-service-ca\") pod \"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf\" (UID: \"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf\") "
Sep 11 14:11:39.953637 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:39.953614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbbf1b1b-bf18-44f3-ad9c-d461e1869baf" (UID: "cbbf1b1b-bf18-44f3-ad9c-d461e1869baf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:11:40.053904 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:40.053875    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:40.704550 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:40.704521    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"
Sep 11 14:11:40.704550 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:40.704540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp" event={"ID":"cbbf1b1b-bf18-44f3-ad9c-d461e1869baf","Type":"ContainerDied","Data":"f99fb2359f85ee8f04b15e861280ef19e809d8f3ad00c7dc5814ddd22215520c"}
Sep 11 14:11:40.704759 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:40.704586    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f99fb2359f85ee8f04b15e861280ef19e809d8f3ad00c7dc5814ddd22215520c"
Sep 11 14:11:47.404885 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.404857    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"]
Sep 11 14:11:47.405282 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.405178    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbf1b1b-bf18-44f3-ad9c-d461e1869baf" containerName="revoke-keys"
Sep 11 14:11:47.405282 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.405192    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbf1b1b-bf18-44f3-ad9c-d461e1869baf" containerName="revoke-keys"
Sep 11 14:11:47.405282 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.405257    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbf1b1b-bf18-44f3-ad9c-d461e1869baf" containerName="revoke-keys"
Sep 11 14:11:47.408050 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.408033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.418415 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.418367    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-fb65d4\""
Sep 11 14:11:47.422758 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.422736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"]
Sep 11 14:11:47.444139 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.444112    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"]
Sep 11 14:11:47.447143 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.447123    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.451143 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.451123    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-fb65d4-dockercfg-9s5k5\""
Sep 11 14:11:47.468350 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.468330    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"]
Sep 11 14:11:47.513853 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.513833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trxk\" (UniqueName: \"kubernetes.io/projected/60b91c86-18ff-4eea-ba02-eefc1294426d-kube-api-access-2trxk\") pod \"payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.513988 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.513861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.513988 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.513901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b91c86-18ff-4eea-ba02-eefc1294426d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.513988 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.513942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qdq\" (UniqueName: \"kubernetes.io/projected/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-kube-api-access-z5qdq\") pod \"payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.614441 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.614412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trxk\" (UniqueName: \"kubernetes.io/projected/60b91c86-18ff-4eea-ba02-eefc1294426d-kube-api-access-2trxk\") pod \"payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.614564 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.614452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.614564 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.614514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b91c86-18ff-4eea-ba02-eefc1294426d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.614564 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.614542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qdq\" (UniqueName: \"kubernetes.io/projected/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-kube-api-access-z5qdq\") pod \"payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.615211 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.615190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.615299 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.615210    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b91c86-18ff-4eea-ba02-eefc1294426d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.630179 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.630154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trxk\" (UniqueName: \"kubernetes.io/projected/60b91c86-18ff-4eea-ba02-eefc1294426d-kube-api-access-2trxk\") pod \"payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.637478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.637459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qdq\" (UniqueName: \"kubernetes.io/projected/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-kube-api-access-z5qdq\") pod \"payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:47.717648 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.717571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:47.755869 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:47.755836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:48.000792 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.000766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"]
Sep 11 14:11:48.002606 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:11:48.002582    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b91c86_18ff_4eea_ba02_eefc1294426d.slice/crio-49e599875fbe62d2b7d32b21dd9a2962bac37b8b9c0230aba377fde4de5fb74b WatchSource:0}: Error finding container 49e599875fbe62d2b7d32b21dd9a2962bac37b8b9c0230aba377fde4de5fb74b: Status 404 returned error can't find the container with id 49e599875fbe62d2b7d32b21dd9a2962bac37b8b9c0230aba377fde4de5fb74b
Sep 11 14:11:48.043812 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.043786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"]
Sep 11 14:11:48.047111 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:11:48.047088    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfe6d6b_49f4_428a_b22d_5f5d36a9363e.slice/crio-297528b67d5fb5ec6c092e8dcdc256d85c6d70c53fad5bca046678997d13d1c2 WatchSource:0}: Error finding container 297528b67d5fb5ec6c092e8dcdc256d85c6d70c53fad5bca046678997d13d1c2: Status 404 returned error can't find the container with id 297528b67d5fb5ec6c092e8dcdc256d85c6d70c53fad5bca046678997d13d1c2
Sep 11 14:11:48.746100 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.746059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx" event={"ID":"60b91c86-18ff-4eea-ba02-eefc1294426d","Type":"ContainerStarted","Data":"15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6"}
Sep 11 14:11:48.746600 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.746399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:11:48.746600 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.746437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx" event={"ID":"60b91c86-18ff-4eea-ba02-eefc1294426d","Type":"ContainerStarted","Data":"49e599875fbe62d2b7d32b21dd9a2962bac37b8b9c0230aba377fde4de5fb74b"}
Sep 11 14:11:48.747744 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.747714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg" event={"ID":"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e","Type":"ContainerStarted","Data":"34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb"}
Sep 11 14:11:48.747881 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.747757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg" event={"ID":"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e","Type":"ContainerStarted","Data":"297528b67d5fb5ec6c092e8dcdc256d85c6d70c53fad5bca046678997d13d1c2"}
Sep 11 14:11:48.747881 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.747787    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:11:48.765018 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.764977    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx" podStartSLOduration=1.7649628480000001 podStartE2EDuration="1.764962848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:48.763165499 +0000 UTC m=+1157.130754942" watchObservedRunningTime="2026-09-11 14:11:48.764962848 +0000 UTC m=+1157.132552290"
Sep 11 14:11:48.782291 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:48.782236    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg" podStartSLOduration=1.782217366 podStartE2EDuration="1.782217366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:48.781521338 +0000 UTC m=+1157.149110782" watchObservedRunningTime="2026-09-11 14:11:48.782217366 +0000 UTC m=+1157.149806813"
Sep 11 14:11:49.429791 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.429655    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"]
Sep 11 14:11:49.430004 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.429921    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm" podUID="a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" containerName="payload-processing" containerID="cri-o://e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7" gracePeriod=30
Sep 11 14:11:49.437791 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.437762    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"]
Sep 11 14:11:49.438047 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.438023    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8" podUID="abe2a114-9c21-45e1-9323-001aee5845b8" containerName="payload-pre-processing" containerID="cri-o://e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c" gracePeriod=30
Sep 11 14:11:49.654766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.654705    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"]
Sep 11 14:11:49.656245 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.656216    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e88320d-89tkp"]
Sep 11 14:11:49.698720 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.698697    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:49.701496 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.701474    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:49.754247 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.754211    2561 generic.go:356] "Generic (PLEG): container finished" podID="a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" containerID="e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7" exitCode=0
Sep 11 14:11:49.754247 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.754243    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"
Sep 11 14:11:49.754745 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.754282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm" event={"ID":"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0","Type":"ContainerDied","Data":"e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7"}
Sep 11 14:11:49.754745 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.754330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm" event={"ID":"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0","Type":"ContainerDied","Data":"b5f3aef58ebbae2cf2d800642115b1b318cd4bf69cf6d966fb422df2264312a5"}
Sep 11 14:11:49.754745 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.754357    2561 scope.go:117] "RemoveContainer" containerID="e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7"
Sep 11 14:11:49.755761 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.755738    2561 generic.go:356] "Generic (PLEG): container finished" podID="abe2a114-9c21-45e1-9323-001aee5845b8" containerID="e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c" exitCode=0
Sep 11 14:11:49.755873 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.755789    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"
Sep 11 14:11:49.755873 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.755836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8" event={"ID":"abe2a114-9c21-45e1-9323-001aee5845b8","Type":"ContainerDied","Data":"e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c"}
Sep 11 14:11:49.756030 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.755878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8" event={"ID":"abe2a114-9c21-45e1-9323-001aee5845b8","Type":"ContainerDied","Data":"9794eaeea6574d42af17d4412fb3cbb95339b334f74f16021685a48a064f8c6a"}
Sep 11 14:11:49.762701 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.762686    2561 scope.go:117] "RemoveContainer" containerID="e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7"
Sep 11 14:11:49.763018 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:49.762997    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7\": container with ID starting with e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7 not found: ID does not exist" containerID="e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7"
Sep 11 14:11:49.763094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.763026    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7"} err="failed to get container status \"e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7\": rpc error: code = NotFound desc = could not find container \"e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7\": container with ID starting with e07050f48360c36ec11aaad5877b2fb380b2ee62fb12b9c3194980a0ef495ef7 not found: ID does not exist"
Sep 11 14:11:49.763094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.763043    2561 scope.go:117] "RemoveContainer" containerID="e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c"
Sep 11 14:11:49.771406 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.771389    2561 scope.go:117] "RemoveContainer" containerID="e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c"
Sep 11 14:11:49.771695 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:11:49.771676    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c\": container with ID starting with e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c not found: ID does not exist" containerID="e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c"
Sep 11 14:11:49.771757 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.771700    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c"} err="failed to get container status \"e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c\": rpc error: code = NotFound desc = could not find container \"e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c\": container with ID starting with e1998caddb1dae94a12a48c72652bb619699556cc7ed39a8f555517113c39b4c not found: ID does not exist"
Sep 11 14:11:49.832921 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.832880    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bnnz\" (UniqueName: \"kubernetes.io/projected/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-kube-api-access-8bnnz\") pod \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") "
Sep 11 14:11:49.833087 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.832941    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe2a114-9c21-45e1-9323-001aee5845b8-plugins-config-volume\") pod \"abe2a114-9c21-45e1-9323-001aee5845b8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") "
Sep 11 14:11:49.833087 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.832970    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-plugins-config-volume\") pod \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\" (UID: \"a7d7de3d-e326-4d4b-b972-e81bfd23f2e0\") "
Sep 11 14:11:49.833217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.833086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szr7q\" (UniqueName: \"kubernetes.io/projected/abe2a114-9c21-45e1-9323-001aee5845b8-kube-api-access-szr7q\") pod \"abe2a114-9c21-45e1-9323-001aee5845b8\" (UID: \"abe2a114-9c21-45e1-9323-001aee5845b8\") "
Sep 11 14:11:49.833633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.833332    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe2a114-9c21-45e1-9323-001aee5845b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abe2a114-9c21-45e1-9323-001aee5845b8" (UID: "abe2a114-9c21-45e1-9323-001aee5845b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:11:49.833633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.833451    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe2a114-9c21-45e1-9323-001aee5845b8-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:49.833633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.833333    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" (UID: "a7d7de3d-e326-4d4b-b972-e81bfd23f2e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:11:49.835261 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.835240    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe2a114-9c21-45e1-9323-001aee5845b8-kube-api-access-szr7q" (OuterVolumeSpecName: "kube-api-access-szr7q") pod "abe2a114-9c21-45e1-9323-001aee5845b8" (UID: "abe2a114-9c21-45e1-9323-001aee5845b8"). InnerVolumeSpecName "kube-api-access-szr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:49.835358 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.835332    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-kube-api-access-8bnnz" (OuterVolumeSpecName: "kube-api-access-8bnnz") pod "a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" (UID: "a7d7de3d-e326-4d4b-b972-e81bfd23f2e0"). InnerVolumeSpecName "kube-api-access-8bnnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:11:49.934677 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.934599    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bnnz\" (UniqueName: \"kubernetes.io/projected/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-kube-api-access-8bnnz\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:49.934677 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.934630    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:49.934677 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:49.934644    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szr7q\" (UniqueName: \"kubernetes.io/projected/abe2a114-9c21-45e1-9323-001aee5845b8-kube-api-access-szr7q\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:11:50.109092 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:50.109064    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"]
Sep 11 14:11:50.123021 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:50.122990    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbf1b1b-bf18-44f3-ad9c-d461e1869baf" path="/var/lib/kubelet/pods/cbbf1b1b-bf18-44f3-ad9c-d461e1869baf/volumes"
Sep 11 14:11:50.123318 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:50.123305    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e88320d-9dd4cd9d6-htgf8"]
Sep 11 14:11:50.138554 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:50.138529    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"]
Sep 11 14:11:50.143685 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:50.143659    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e88320d-5dd8646bbf-cgcnm"]
Sep 11 14:11:51.587596 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:51.587561    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:11:52.116574 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:52.116542    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" path="/var/lib/kubelet/pods/a7d7de3d-e326-4d4b-b972-e81bfd23f2e0/volumes"
Sep 11 14:11:52.116924 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:52.116898    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe2a114-9c21-45e1-9323-001aee5845b8" path="/var/lib/kubelet/pods/abe2a114-9c21-45e1-9323-001aee5845b8/volumes"
Sep 11 14:11:59.758299 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:11:59.758221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:12:00.764486 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:00.764454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:12:12.220938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.220888    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"]
Sep 11 14:12:12.221365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.221192    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" containerName="payload-processing"
Sep 11 14:12:12.221365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.221208    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" containerName="payload-processing"
Sep 11 14:12:12.221365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.221223    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe2a114-9c21-45e1-9323-001aee5845b8" containerName="payload-pre-processing"
Sep 11 14:12:12.221365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.221228    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe2a114-9c21-45e1-9323-001aee5845b8" containerName="payload-pre-processing"
Sep 11 14:12:12.221365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.221287    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d7de3d-e326-4d4b-b972-e81bfd23f2e0" containerName="payload-processing"
Sep 11 14:12:12.221365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.221297    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe2a114-9c21-45e1-9323-001aee5845b8" containerName="payload-pre-processing"
Sep 11 14:12:12.224158 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.224141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.228239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.228220    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:12:12.228465 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.228445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c52865-openshift-default-dockercfg-jwgvr\""
Sep 11 14:12:12.240973 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.240940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"]
Sep 11 14:12:12.313716 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-credential-socket\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.313716 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-socket\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-certs\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-podinfo\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rbx\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-kube-api-access-l2rbx\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-envoy\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.313973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-data\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.314023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istiod-ca-cert\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.314283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.314051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-token\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415006 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.414974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istiod-ca-cert\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415006 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-token\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-credential-socket\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-socket\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-certs\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-podinfo\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rbx\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-kube-api-access-l2rbx\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-envoy\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-data\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415640 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-socket\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415640 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-credential-socket\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415780 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istiod-ca-cert\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415846 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-data\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.415902 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.415886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-certs\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.417548 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.417525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-envoy\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.417737 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.417709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-podinfo\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.423763 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.423743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-token\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.427883 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.427864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rbx\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-kube-api-access-l2rbx\") pod \"e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") " pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.533832 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.533740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:12.820603 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.820568    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"]
Sep 11 14:12:12.823733 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:12:12.823704    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f94f2f2_c2de_4750_8998_551fd21bf9d4.slice/crio-01760fcc7f73f0f477fcb4844c7a82fb2cc1457c0caf2a40b392f7c0f4f3e913 WatchSource:0}: Error finding container 01760fcc7f73f0f477fcb4844c7a82fb2cc1457c0caf2a40b392f7c0f4f3e913: Status 404 returned error can't find the container with id 01760fcc7f73f0f477fcb4844c7a82fb2cc1457c0caf2a40b392f7c0f4f3e913
Sep 11 14:12:12.825850 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.825817    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:12:12.825933 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.825886    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:12:12.825978 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.825933    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:12:12.867578 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:12.867548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" event={"ID":"7f94f2f2-c2de-4750-8998-551fd21bf9d4","Type":"ContainerStarted","Data":"01760fcc7f73f0f477fcb4844c7a82fb2cc1457c0caf2a40b392f7c0f4f3e913"}
Sep 11 14:12:13.873003 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:13.872966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" event={"ID":"7f94f2f2-c2de-4750-8998-551fd21bf9d4","Type":"ContainerStarted","Data":"42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"}
Sep 11 14:12:13.899311 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:13.899261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podStartSLOduration=1.8992459670000001 podStartE2EDuration="1.899245967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:13.898061281 +0000 UTC m=+1182.265650723" watchObservedRunningTime="2026-09-11 14:12:13.899245967 +0000 UTC m=+1182.266835410"
Sep 11 14:12:14.534930 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:14.534880    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:14.539723 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:14.539689    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:14.877864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:14.877838    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:14.879000 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:14.878973    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:16.697665 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.697630    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"]
Sep 11 14:12:16.700963 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.700944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.703803 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.703779    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-261f9c\""
Sep 11 14:12:16.713288 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.713264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"]
Sep 11 14:12:16.749738 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.749708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b847e943-f370-4e26-98f1-3f96bd84a884-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.749903 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.749748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgbn\" (UniqueName: \"kubernetes.io/projected/b847e943-f370-4e26-98f1-3f96bd84a884-kube-api-access-pfgbn\") pod \"payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.850370 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.850333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b847e943-f370-4e26-98f1-3f96bd84a884-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.850370 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.850377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgbn\" (UniqueName: \"kubernetes.io/projected/b847e943-f370-4e26-98f1-3f96bd84a884-kube-api-access-pfgbn\") pod \"payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.851112 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.851092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b847e943-f370-4e26-98f1-3f96bd84a884-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.855270 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.855242    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"]
Sep 11 14:12:16.858496 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.858477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:16.860593 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.860567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgbn\" (UniqueName: \"kubernetes.io/projected/b847e943-f370-4e26-98f1-3f96bd84a884-kube-api-access-pfgbn\") pod \"payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:16.861058 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.861037    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-261f9c-dockercfg-wzmd5\""
Sep 11 14:12:16.872686 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.872659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"]
Sep 11 14:12:16.951538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.951459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xht5f\" (UniqueName: \"kubernetes.io/projected/5c2af385-e8ff-4d9f-a86d-051290c30ccf-kube-api-access-xht5f\") pod \"payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:16.951700 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:16.951581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2af385-e8ff-4d9f-a86d-051290c30ccf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.010771 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.010748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:17.052420 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.052387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xht5f\" (UniqueName: \"kubernetes.io/projected/5c2af385-e8ff-4d9f-a86d-051290c30ccf-kube-api-access-xht5f\") pod \"payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.052557 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.052471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2af385-e8ff-4d9f-a86d-051290c30ccf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.053130 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.053111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2af385-e8ff-4d9f-a86d-051290c30ccf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.068400 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.068371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xht5f\" (UniqueName: \"kubernetes.io/projected/5c2af385-e8ff-4d9f-a86d-051290c30ccf-kube-api-access-xht5f\") pod \"payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.175978 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.175946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.291670 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.291639    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"]
Sep 11 14:12:17.294796 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:12:17.294774    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb847e943_f370_4e26_98f1_3f96bd84a884.slice/crio-8c51c6ce67e9b37960afac1401782b02372f382fbed504fd69887afac7a85090 WatchSource:0}: Error finding container 8c51c6ce67e9b37960afac1401782b02372f382fbed504fd69887afac7a85090: Status 404 returned error can't find the container with id 8c51c6ce67e9b37960afac1401782b02372f382fbed504fd69887afac7a85090
Sep 11 14:12:17.469489 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.469456    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"]
Sep 11 14:12:17.474005 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:12:17.473954    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2af385_e8ff_4d9f_a86d_051290c30ccf.slice/crio-f38bf494231b6afb6f8dcf12185a014a37a449581f8c99077d02061aae36b83e WatchSource:0}: Error finding container f38bf494231b6afb6f8dcf12185a014a37a449581f8c99077d02061aae36b83e: Status 404 returned error can't find the container with id f38bf494231b6afb6f8dcf12185a014a37a449581f8c99077d02061aae36b83e
Sep 11 14:12:17.893741 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.893702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj" event={"ID":"5c2af385-e8ff-4d9f-a86d-051290c30ccf","Type":"ContainerStarted","Data":"310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72"}
Sep 11 14:12:17.894216 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.893762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj" event={"ID":"5c2af385-e8ff-4d9f-a86d-051290c30ccf","Type":"ContainerStarted","Data":"f38bf494231b6afb6f8dcf12185a014a37a449581f8c99077d02061aae36b83e"}
Sep 11 14:12:17.894216 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.893788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:17.895348 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.895321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr" event={"ID":"b847e943-f370-4e26-98f1-3f96bd84a884","Type":"ContainerStarted","Data":"68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574"}
Sep 11 14:12:17.895462 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.895363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr" event={"ID":"b847e943-f370-4e26-98f1-3f96bd84a884","Type":"ContainerStarted","Data":"8c51c6ce67e9b37960afac1401782b02372f382fbed504fd69887afac7a85090"}
Sep 11 14:12:17.895462 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.895409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:17.918856 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.918810    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj" podStartSLOduration=1.918796338 podStartE2EDuration="1.918796338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:17.918091334 +0000 UTC m=+1186.285680776" watchObservedRunningTime="2026-09-11 14:12:17.918796338 +0000 UTC m=+1186.286385780"
Sep 11 14:12:17.964313 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:17.964272    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr" podStartSLOduration=1.964258208 podStartE2EDuration="1.964258208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:17.964005706 +0000 UTC m=+1186.331595152" watchObservedRunningTime="2026-09-11 14:12:17.964258208 +0000 UTC m=+1186.331847650"
Sep 11 14:12:20.199506 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:20.199471    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:12:21.015193 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.015161    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"]
Sep 11 14:12:21.019551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.019533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.024788 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.024760    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c52865-dockercfg-lv6xr\""
Sep 11 14:12:21.024874 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.024785    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c52865\""
Sep 11 14:12:21.037300 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.037264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"]
Sep 11 14:12:21.085946 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.085893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txtj\" (UniqueName: \"kubernetes.io/projected/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-kube-api-access-2txtj\") pod \"payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.086106 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.086022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.186605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.186563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txtj\" (UniqueName: \"kubernetes.io/projected/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-kube-api-access-2txtj\") pod \"payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.186766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.186656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.187398 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.187378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.211100 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.211065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txtj\" (UniqueName: \"kubernetes.io/projected/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-kube-api-access-2txtj\") pod \"payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.331412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.331371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.632106 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.632079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"]
Sep 11 14:12:21.633970 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:12:21.633938    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9115c6ef_7ed7_4c8b_b62e_e5da49684d47.slice/crio-95bf7d5f78fc512876bdfbc51f1267515d4945a719d1b429315c1e45908cc826 WatchSource:0}: Error finding container 95bf7d5f78fc512876bdfbc51f1267515d4945a719d1b429315c1e45908cc826: Status 404 returned error can't find the container with id 95bf7d5f78fc512876bdfbc51f1267515d4945a719d1b429315c1e45908cc826
Sep 11 14:12:21.915079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.915000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4" event={"ID":"9115c6ef-7ed7-4c8b-b62e-e5da49684d47","Type":"ContainerStarted","Data":"3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c"}
Sep 11 14:12:21.915079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.915050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4" event={"ID":"9115c6ef-7ed7-4c8b-b62e-e5da49684d47","Type":"ContainerStarted","Data":"95bf7d5f78fc512876bdfbc51f1267515d4945a719d1b429315c1e45908cc826"}
Sep 11 14:12:21.915267 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.915142    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:21.940270 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:21.940222    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4" podStartSLOduration=1.940202765 podStartE2EDuration="1.940202765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:21.937822208 +0000 UTC m=+1190.305411651" watchObservedRunningTime="2026-09-11 14:12:21.940202765 +0000 UTC m=+1190.307792209"
Sep 11 14:12:28.901535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:28.901498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:12:28.902050 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:28.901926    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:12:30.693330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:30.693301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:12:32.149246 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:32.149214    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:12:32.151460 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:32.151438    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:12:32.921591 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:32.921557    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:12:33.979760 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:33.979670    2561 generic.go:356] "Generic (PLEG): container finished" podID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerID="42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41" exitCode=0
Sep 11 14:12:33.979760 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:33.979730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" event={"ID":"7f94f2f2-c2de-4750-8998-551fd21bf9d4","Type":"ContainerDied","Data":"42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"}
Sep 11 14:12:33.980283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:33.980113    2561 scope.go:117] "RemoveContainer" containerID="42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"
Sep 11 14:12:33.981131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:33.981100    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:12:33.981249 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:33.981186    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:12:33.981249 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:33.981220    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:12:34.986389 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:34.986354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" event={"ID":"7f94f2f2-c2de-4750-8998-551fd21bf9d4","Type":"ContainerStarted","Data":"d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b"}
Sep 11 14:12:35.534131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:35.534085    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:35.534430 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:35.534399    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:12:35.534534 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:35.534467    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 14:12:36.535147 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:36.535101    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:12:36.535659 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:36.535187    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 14:12:37.534596 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:37.534549    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:12:37.534814 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:37.534621    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 14:12:38.534409 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:38.534359    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:12:38.534928 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:38.534434    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 14:12:39.535121 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:39.535074    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:12:39.535650 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:39.535151    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 11 14:12:40.539134 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:40.539100    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:40.539556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:40.539419    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:40.540169 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:40.540146    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:12:53.940142 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:53.940109    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"]
Sep 11 14:12:53.944820 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:53.944798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:53.948804 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:53.948782    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5cf565\""
Sep 11 14:12:53.959247 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:53.959220    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"]
Sep 11 14:12:54.075620 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.075585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9smg\" (UniqueName: \"kubernetes.io/projected/fa5cf026-ae2e-4204-a1be-b67217d18de8-kube-api-access-x9smg\") pod \"payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.075620 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.075627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5cf026-ae2e-4204-a1be-b67217d18de8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.177163 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.177129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9smg\" (UniqueName: \"kubernetes.io/projected/fa5cf026-ae2e-4204-a1be-b67217d18de8-kube-api-access-x9smg\") pod \"payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.177349 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.177185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5cf026-ae2e-4204-a1be-b67217d18de8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.177950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.177932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5cf026-ae2e-4204-a1be-b67217d18de8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.188678 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.188652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9smg\" (UniqueName: \"kubernetes.io/projected/fa5cf026-ae2e-4204-a1be-b67217d18de8-kube-api-access-x9smg\") pod \"payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.254565 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.254503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:54.527171 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:54.527096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"]
Sep 11 14:12:54.530229 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:12:54.530182    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5cf026_ae2e_4204_a1be_b67217d18de8.slice/crio-3d969f91ed9f487fec4819a8417bcdfb1b97bfcd5ade2bb3d73c0bf7292973b9 WatchSource:0}: Error finding container 3d969f91ed9f487fec4819a8417bcdfb1b97bfcd5ade2bb3d73c0bf7292973b9: Status 404 returned error can't find the container with id 3d969f91ed9f487fec4819a8417bcdfb1b97bfcd5ade2bb3d73c0bf7292973b9
Sep 11 14:12:55.081704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:55.081668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n" event={"ID":"fa5cf026-ae2e-4204-a1be-b67217d18de8","Type":"ContainerStarted","Data":"588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22"}
Sep 11 14:12:55.082231 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:55.081715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n" event={"ID":"fa5cf026-ae2e-4204-a1be-b67217d18de8","Type":"ContainerStarted","Data":"3d969f91ed9f487fec4819a8417bcdfb1b97bfcd5ade2bb3d73c0bf7292973b9"}
Sep 11 14:12:55.082231 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:55.081745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:12:55.106237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:55.106183    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n" podStartSLOduration=2.106167601 podStartE2EDuration="2.106167601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:55.104808538 +0000 UTC m=+1223.472397978" watchObservedRunningTime="2026-09-11 14:12:55.106167601 +0000 UTC m=+1223.473757045"
Sep 11 14:12:58.122217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.122183    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"]
Sep 11 14:12:58.125587 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.125563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.135486 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.135461    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"auth-test-model-fb65d4-kserve-self-signed-certs\""
Sep 11 14:12:58.136515 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.136493    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"openshift-service-ca.crt\""
Sep 11 14:12:58.136591 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.136537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"default-dockercfg-gchl6\""
Sep 11 14:12:58.137866 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.137845    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"kube-root-ca.crt\""
Sep 11 14:12:58.149769 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.149744    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:12:58.199042 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.199011    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"]
Sep 11 14:12:58.316924 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.316875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-home\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.317109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.316961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-dshm\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.317109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.317017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-tmp-dir\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.317109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.317075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-model-cache\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.317372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.317129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80447a04-cb48-42c7-931b-fde90370f062-tls-certs\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.317372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.317152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp88t\" (UniqueName: \"kubernetes.io/projected/80447a04-cb48-42c7-931b-fde90370f062-kube-api-access-rp88t\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.418623 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.418540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-tmp-dir\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.418623 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.418591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-model-cache\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.418623 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.418624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80447a04-cb48-42c7-931b-fde90370f062-tls-certs\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.418872 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.418640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp88t\" (UniqueName: \"kubernetes.io/projected/80447a04-cb48-42c7-931b-fde90370f062-kube-api-access-rp88t\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.418872 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.418672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-home\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.418872 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.418713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-dshm\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.419066 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.419019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-tmp-dir\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.419152 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.419134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-model-cache\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.419215 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.419188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-home\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.420928 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.420895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-dshm\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.421166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.421152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80447a04-cb48-42c7-931b-fde90370f062-tls-certs\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.435538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.435516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp88t\" (UniqueName: \"kubernetes.io/projected/80447a04-cb48-42c7-931b-fde90370f062-kube-api-access-rp88t\") pod \"auth-test-model-fb65d4-kserve-79586b6c6c-p6675\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.437369 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.437348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:12:58.723454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:58.723366    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"]
Sep 11 14:12:58.726158 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:12:58.726127    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80447a04_cb48_42c7_931b_fde90370f062.slice/crio-ab5b7c28d4cdc3d5a239d6629baa3cc20062ea41377be8bb2e3a1602a59b9e11 WatchSource:0}: Error finding container ab5b7c28d4cdc3d5a239d6629baa3cc20062ea41377be8bb2e3a1602a59b9e11: Status 404 returned error can't find the container with id ab5b7c28d4cdc3d5a239d6629baa3cc20062ea41377be8bb2e3a1602a59b9e11
Sep 11 14:12:59.102523 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:59.102486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675" event={"ID":"80447a04-cb48-42c7-931b-fde90370f062","Type":"ContainerStarted","Data":"3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7"}
Sep 11 14:12:59.102735 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:59.102535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675" event={"ID":"80447a04-cb48-42c7-931b-fde90370f062","Type":"ContainerStarted","Data":"ab5b7c28d4cdc3d5a239d6629baa3cc20062ea41377be8bb2e3a1602a59b9e11"}
Sep 11 14:12:59.128160 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:12:59.128112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675" podStartSLOduration=1.128096776 podStartE2EDuration="1.128096776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:59.127791164 +0000 UTC m=+1227.495380606" watchObservedRunningTime="2026-09-11 14:12:59.128096776 +0000 UTC m=+1227.495686220"
Sep 11 14:13:06.089847 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:06.089815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:13:08.438162 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:08.438120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:13:08.438537 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:08.438380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:13:08.450649 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:08.450621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:13:09.158527 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:09.158497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:13:10.707870 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:10.707837    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:13:17.032061 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.032028    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"]
Sep 11 14:13:17.035150 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.035134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.038490 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:17.038460    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-qjcj7\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w4-261f9c\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"default-dockercfg-qjcj7\"" type="*v1.Secret"
Sep 11 14:13:17.038760 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.038738    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"openshift-service-ca.crt\""
Sep 11 14:13:17.038832 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.038770    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"kube-root-ca.crt\""
Sep 11 14:13:17.040276 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:17.040254    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"auth-test-model-261f9c-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w4-261f9c\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"auth-test-model-261f9c-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 11 14:13:17.040357 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:17.040246    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-w4-261f9c\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="1960bc53-76b1-4012-95ca-37d7e7361ffa" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.071188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.071154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.071188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.071198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-tmp-dir\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.071392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.071233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-dshm\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.071392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.071251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-model-cache\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.071392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.071294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-home\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.071392 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.071366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrqs\" (UniqueName: \"kubernetes.io/projected/1960bc53-76b1-4012-95ca-37d7e7361ffa-kube-api-access-vfrqs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.094003 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.093969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"]
Sep 11 14:13:17.172016 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.171980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrqs\" (UniqueName: \"kubernetes.io/projected/1960bc53-76b1-4012-95ca-37d7e7361ffa-kube-api-access-vfrqs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172016 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-tmp-dir\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-dshm\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-model-cache\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-home\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172494 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-tmp-dir\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172532 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-model-cache\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.172566 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.172540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-home\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.174456 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.174437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-dshm\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.189150 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.189126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrqs\" (UniqueName: \"kubernetes.io/projected/1960bc53-76b1-4012-95ca-37d7e7361ffa-kube-api-access-vfrqs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:17.906437 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:17.906407    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"default-dockercfg-qjcj7\""
Sep 11 14:13:18.172994 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:18.172878    2561 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 11 14:13:18.172994 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:18.172969    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs podName:1960bc53-76b1-4012-95ca-37d7e7361ffa nodeName:}" failed. No retries permitted until 2026-09-11 14:13:18.672948958 +0000 UTC m=+1247.040538386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs") pod "auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa") : failed to sync secret cache: timed out waiting for the condition
Sep 11 14:13:18.312670 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:18.312644    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"auth-test-model-261f9c-kserve-self-signed-certs\""
Sep 11 14:13:18.685029 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:18.684999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:18.687936 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:18.687892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs\") pod \"auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:18.845011 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:18.844978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:19.127021 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:19.126813    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"]
Sep 11 14:13:19.129628 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:13:19.129602    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1960bc53_76b1_4012_95ca_37d7e7361ffa.slice/crio-71459cdd67560d0e79e81366ba6af18e78fe1c02eaa9b4187d2725a074f0f970 WatchSource:0}: Error finding container 71459cdd67560d0e79e81366ba6af18e78fe1c02eaa9b4187d2725a074f0f970: Status 404 returned error can't find the container with id 71459cdd67560d0e79e81366ba6af18e78fe1c02eaa9b4187d2725a074f0f970
Sep 11 14:13:19.194255 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:19.194224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" event={"ID":"1960bc53-76b1-4012-95ca-37d7e7361ffa","Type":"ContainerStarted","Data":"5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372"}
Sep 11 14:13:19.194539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:19.194268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" event={"ID":"1960bc53-76b1-4012-95ca-37d7e7361ffa","Type":"ContainerStarted","Data":"71459cdd67560d0e79e81366ba6af18e78fe1c02eaa9b4187d2725a074f0f970"}
Sep 11 14:13:20.232688 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.232628    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" podStartSLOduration=4.232611106 podStartE2EDuration="4.232611106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:20.232022291 +0000 UTC m=+1248.599611735" watchObservedRunningTime="2026-09-11 14:13:20.232611106 +0000 UTC m=+1248.600200549"
Sep 11 14:13:20.468588 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.468557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7743-4d50d3"]
Sep 11 14:13:20.471823 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.471805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:20.475073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.475052    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m8d75\""
Sep 11 14:13:20.485186 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.485123    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-4d50d3"]
Sep 11 14:13:20.600491 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.600453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fqf\" (UniqueName: \"kubernetes.io/projected/d592f01f-d09a-48a6-ad4f-2b03955f70eb-kube-api-access-v4fqf\") pod \"test-curl-7743-4d50d3\" (UID: \"d592f01f-d09a-48a6-ad4f-2b03955f70eb\") " pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:20.701312 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.701277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fqf\" (UniqueName: \"kubernetes.io/projected/d592f01f-d09a-48a6-ad4f-2b03955f70eb-kube-api-access-v4fqf\") pod \"test-curl-7743-4d50d3\" (UID: \"d592f01f-d09a-48a6-ad4f-2b03955f70eb\") " pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:20.714889 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.714871    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fqf\" (UniqueName: \"kubernetes.io/projected/d592f01f-d09a-48a6-ad4f-2b03955f70eb-kube-api-access-v4fqf\") pod \"test-curl-7743-4d50d3\" (UID: \"d592f01f-d09a-48a6-ad4f-2b03955f70eb\") " pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:20.779893 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:20.779817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:21.052828 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.052768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-4d50d3"]
Sep 11 14:13:21.057097 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:13:21.057067    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd592f01f_d09a_48a6_ad4f_2b03955f70eb.slice/crio-c0598daf609ab29dfd13dbc4d59ce784a56e18018ae1af662e3af022508a8656 WatchSource:0}: Error finding container c0598daf609ab29dfd13dbc4d59ce784a56e18018ae1af662e3af022508a8656: Status 404 returned error can't find the container with id c0598daf609ab29dfd13dbc4d59ce784a56e18018ae1af662e3af022508a8656
Sep 11 14:13:21.103211 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.103186    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:13:21.204922 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.204879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-4d50d3" event={"ID":"d592f01f-d09a-48a6-ad4f-2b03955f70eb","Type":"ContainerStarted","Data":"168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f"}
Sep 11 14:13:21.205021 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.204945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-4d50d3" event={"ID":"d592f01f-d09a-48a6-ad4f-2b03955f70eb","Type":"ContainerStarted","Data":"c0598daf609ab29dfd13dbc4d59ce784a56e18018ae1af662e3af022508a8656"}
Sep 11 14:13:21.224094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.224053    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7743-4d50d3" podStartSLOduration=1.224038464 podStartE2EDuration="1.224038464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:21.222195021 +0000 UTC m=+1249.589784464" watchObservedRunningTime="2026-09-11 14:13:21.224038464 +0000 UTC m=+1249.591627908"
Sep 11 14:13:21.822960 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.822934    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7743-4d50d3"]
Sep 11 14:13:21.828153 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:21.828122    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7743-4d50d3"]
Sep 11 14:13:22.209461 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:22.209380    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7743-4d50d3" container="test-curl-7743-4d50d3"
Sep 11 14:13:23.219715 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:23.219670    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7743-4d50d3" podUID="d592f01f-d09a-48a6-ad4f-2b03955f70eb" containerName="test-curl-7743-4d50d3" containerID="cri-o://168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f" gracePeriod=2
Sep 11 14:13:25.395759 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:25.395735    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:25.437963 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:25.437936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4fqf\" (UniqueName: \"kubernetes.io/projected/d592f01f-d09a-48a6-ad4f-2b03955f70eb-kube-api-access-v4fqf\") pod \"d592f01f-d09a-48a6-ad4f-2b03955f70eb\" (UID: \"d592f01f-d09a-48a6-ad4f-2b03955f70eb\") "
Sep 11 14:13:25.439858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:25.439835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d592f01f-d09a-48a6-ad4f-2b03955f70eb-kube-api-access-v4fqf" (OuterVolumeSpecName: "kube-api-access-v4fqf") pod "d592f01f-d09a-48a6-ad4f-2b03955f70eb" (UID: "d592f01f-d09a-48a6-ad4f-2b03955f70eb"). InnerVolumeSpecName "kube-api-access-v4fqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:13:25.538902 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:25.538840    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fqf\" (UniqueName: \"kubernetes.io/projected/d592f01f-d09a-48a6-ad4f-2b03955f70eb-kube-api-access-v4fqf\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:13:26.117242 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:26.117205    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d592f01f-d09a-48a6-ad4f-2b03955f70eb" path="/var/lib/kubelet/pods/d592f01f-d09a-48a6-ad4f-2b03955f70eb/volumes"
Sep 11 14:13:26.234945 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:26.234890    2561 generic.go:356] "Generic (PLEG): container finished" podID="d592f01f-d09a-48a6-ad4f-2b03955f70eb" containerID="168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f" exitCode=137
Sep 11 14:13:26.234945 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:26.234946    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-4d50d3"
Sep 11 14:13:26.235177 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:26.234966    2561 scope.go:117] "RemoveContainer" containerID="168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f"
Sep 11 14:13:26.243061 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:26.243026    2561 scope.go:117] "RemoveContainer" containerID="168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f"
Sep 11 14:13:26.243318 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:13:26.243300    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f\": container with ID starting with 168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f not found: ID does not exist" containerID="168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f"
Sep 11 14:13:26.243383 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:26.243330    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f"} err="failed to get container status \"168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f\": rpc error: code = NotFound desc = could not find container \"168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f\": container with ID starting with 168c71bb11e6a488559f6f9c5b3b2e3efd162146091daac95ddf1ac470c42e7f not found: ID does not exist"
Sep 11 14:13:28.845812 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:28.845779    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:28.846280 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:28.845825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:28.858174 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:28.858153    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:29.260151 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:29.260066    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:13:32.241557 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:32.241523    2561 scope.go:117] "RemoveContainer" containerID="fab6bd2fe7be4d3cf62ae139dbb7204b08b4191ebdb87ab228da008916ebfbd1"
Sep 11 14:13:39.704892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.704856    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"]
Sep 11 14:13:39.705426 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.705404    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d592f01f-d09a-48a6-ad4f-2b03955f70eb" containerName="test-curl-7743-4d50d3"
Sep 11 14:13:39.705524 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.705430    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d592f01f-d09a-48a6-ad4f-2b03955f70eb" containerName="test-curl-7743-4d50d3"
Sep 11 14:13:39.705595 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.705528    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d592f01f-d09a-48a6-ad4f-2b03955f70eb" containerName="test-curl-7743-4d50d3"
Sep 11 14:13:39.710272 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.710241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:39.716435 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.716402    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d630fb00-dockercfg-v2pmw\""
Sep 11 14:13:39.716778 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.716755    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d630fb00\""
Sep 11 14:13:39.731636 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.731606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"]
Sep 11 14:13:39.871643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.871601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljt4c\" (UniqueName: \"kubernetes.io/projected/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-kube-api-access-ljt4c\") pod \"payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:39.872027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.871999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:39.973344 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.973232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljt4c\" (UniqueName: \"kubernetes.io/projected/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-kube-api-access-ljt4c\") pod \"payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:39.973522 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.973359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:39.974327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.974302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:39.984815 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:39.984785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljt4c\" (UniqueName: \"kubernetes.io/projected/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-kube-api-access-ljt4c\") pod \"payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:40.025997 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:40.025967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:40.333302 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:40.331667    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"]
Sep 11 14:13:40.338899 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:13:40.338866    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4df65fb_c71c_40fa_b9c1_ccfd6b3c3194.slice/crio-e8c524a7b4b8855649d54c61eaacfae45874a97b2dc559dbcc7fc03f63125a55 WatchSource:0}: Error finding container e8c524a7b4b8855649d54c61eaacfae45874a97b2dc559dbcc7fc03f63125a55: Status 404 returned error can't find the container with id e8c524a7b4b8855649d54c61eaacfae45874a97b2dc559dbcc7fc03f63125a55
Sep 11 14:13:41.316151 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:41.316119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf" event={"ID":"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194","Type":"ContainerStarted","Data":"ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3"}
Sep 11 14:13:41.316525 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:41.316172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf" event={"ID":"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194","Type":"ContainerStarted","Data":"e8c524a7b4b8855649d54c61eaacfae45874a97b2dc559dbcc7fc03f63125a55"}
Sep 11 14:13:41.316525 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:41.316244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:41.339651 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:41.339594    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf" podStartSLOduration=2.339576045 podStartE2EDuration="2.339576045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:41.338181653 +0000 UTC m=+1269.705771097" watchObservedRunningTime="2026-09-11 14:13:41.339576045 +0000 UTC m=+1269.707165487"
Sep 11 14:13:46.806337 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:46.806301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:13:52.324092 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:52.324060    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:13:59.434762 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:13:59.434729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:14:08.804802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.804766    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"]
Sep 11 14:14:08.808130 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.808108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:08.813053 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.813030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d630fb00\"/\"openshift-service-ca.crt\""
Sep 11 14:14:08.823586 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.823568    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d630fb00\"/\"kube-root-ca.crt\""
Sep 11 14:14:08.823852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.823837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d630fb00\"/\"default-dockercfg-fj8cd\""
Sep 11 14:14:08.824170 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.824156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d630fb00\"/\"e2e-lifecycle-model-d630fb00-kserve-self-signed-certs\""
Sep 11 14:14:08.830393 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.830363    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"]
Sep 11 14:14:08.907055 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.907023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tls-certs\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:08.907202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.907060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwkt\" (UniqueName: \"kubernetes.io/projected/20ee64a0-ca84-40f1-9b42-9f91369bf10b-kube-api-access-hgwkt\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:08.907202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.907089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tmp-dir\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:08.907289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.907196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-home\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:08.907289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.907229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-dshm\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:08.907289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:08.907256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-model-cache\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008532 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-home\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-dshm\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-model-cache\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tls-certs\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008901 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwkt\" (UniqueName: \"kubernetes.io/projected/20ee64a0-ca84-40f1-9b42-9f91369bf10b-kube-api-access-hgwkt\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008901 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tmp-dir\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.008901 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-home\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.009078 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.008981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-model-cache\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.009181 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.009163    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tmp-dir\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.010961 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.010933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-dshm\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.011130 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.011115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tls-certs\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.024042 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.024022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwkt\" (UniqueName: \"kubernetes.io/projected/20ee64a0-ca84-40f1-9b42-9f91369bf10b-kube-api-access-hgwkt\") pod \"e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") " pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.117376 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.117351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:09.405156 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.405126    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"]
Sep 11 14:14:09.406615 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:14:09.406590    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20ee64a0_ca84_40f1_9b42_9f91369bf10b.slice/crio-44bcbba689852604b64de9b4ccd14c9abe60f0ae05ffbd6b48adf412d42e70a9 WatchSource:0}: Error finding container 44bcbba689852604b64de9b4ccd14c9abe60f0ae05ffbd6b48adf412d42e70a9: Status 404 returned error can't find the container with id 44bcbba689852604b64de9b4ccd14c9abe60f0ae05ffbd6b48adf412d42e70a9
Sep 11 14:14:09.447164 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:09.447135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255" event={"ID":"20ee64a0-ca84-40f1-9b42-9f91369bf10b","Type":"ContainerStarted","Data":"44bcbba689852604b64de9b4ccd14c9abe60f0ae05ffbd6b48adf412d42e70a9"}
Sep 11 14:14:10.453057 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:10.453013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255" event={"ID":"20ee64a0-ca84-40f1-9b42-9f91369bf10b","Type":"ContainerStarted","Data":"df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070"}
Sep 11 14:14:10.480487 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:10.480436    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255" podStartSLOduration=2.480421576 podStartE2EDuration="2.480421576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:10.478779715 +0000 UTC m=+1298.846369158" watchObservedRunningTime="2026-09-11 14:14:10.480421576 +0000 UTC m=+1298.848011016"
Sep 11 14:14:15.782657 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:15.782619    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:14:19.118191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:19.118136    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:19.118191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:19.118204    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:19.130711 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:19.130687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:19.507070 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:19.506990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:14:32.538614 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:32.538580    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:14:50.707631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:50.707600    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:14:54.050300 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.050264    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"]
Sep 11 14:14:54.050772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.050532    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf" podUID="f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" containerName="payload-pre-processing" containerID="cri-o://ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3" gracePeriod=30
Sep 11 14:14:54.298551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.298527    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:14:54.300636 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.300601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-plugins-config-volume\") pod \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") "
Sep 11 14:14:54.300693 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.300648    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljt4c\" (UniqueName: \"kubernetes.io/projected/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-kube-api-access-ljt4c\") pod \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\" (UID: \"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194\") "
Sep 11 14:14:54.300946 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.300904    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" (UID: "f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:14:54.302737 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.302708    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-kube-api-access-ljt4c" (OuterVolumeSpecName: "kube-api-access-ljt4c") pod "f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" (UID: "f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194"). InnerVolumeSpecName "kube-api-access-ljt4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:14:54.401222 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.401188    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljt4c\" (UniqueName: \"kubernetes.io/projected/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-kube-api-access-ljt4c\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:14:54.401222 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.401221    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:14:54.666010 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.665976    2561 generic.go:356] "Generic (PLEG): container finished" podID="f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" containerID="ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3" exitCode=0
Sep 11 14:14:54.666169 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.666026    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"
Sep 11 14:14:54.666169 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.666053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf" event={"ID":"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194","Type":"ContainerDied","Data":"ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3"}
Sep 11 14:14:54.666169 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.666094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf" event={"ID":"f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194","Type":"ContainerDied","Data":"e8c524a7b4b8855649d54c61eaacfae45874a97b2dc559dbcc7fc03f63125a55"}
Sep 11 14:14:54.666169 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.666116    2561 scope.go:117] "RemoveContainer" containerID="ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3"
Sep 11 14:14:54.674822 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.674805    2561 scope.go:117] "RemoveContainer" containerID="ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3"
Sep 11 14:14:54.675110 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:14:54.675091    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3\": container with ID starting with ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3 not found: ID does not exist" containerID="ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3"
Sep 11 14:14:54.675180 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.675118    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3"} err="failed to get container status \"ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3\": rpc error: code = NotFound desc = could not find container \"ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3\": container with ID starting with ad7b0284e5ce26f30a51006382d57ec1c3ed01ff8c4fa883ff0815ec420e8fc3 not found: ID does not exist"
Sep 11 14:14:54.691832 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.691809    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"]
Sep 11 14:14:54.699313 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:54.699295    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d630fb00-667dc57b66-9vbpf"]
Sep 11 14:14:56.117533 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:14:56.117499    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" path="/var/lib/kubelet/pods/f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194/volumes"
Sep 11 14:15:00.241448 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.241409    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"]
Sep 11 14:15:00.241967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.241856    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" containerName="payload-pre-processing"
Sep 11 14:15:00.241967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.241874    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" containerName="payload-pre-processing"
Sep 11 14:15:00.242104 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.242014    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4df65fb-c71c-40fa-b9c1-ccfd6b3c3194" containerName="payload-pre-processing"
Sep 11 14:15:00.246393 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.246371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"
Sep 11 14:15:00.248773 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.248746    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"]
Sep 11 14:15:00.249948 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.249927    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:15:00.251679 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.251664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"
Sep 11 14:15:00.257244 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.257227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"
Sep 11 14:15:00.260608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.260583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"]
Sep 11 14:15:00.266319 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.266298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"
Sep 11 14:15:00.309799 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.309680    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"]
Sep 11 14:15:00.581226 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:15:00.581184    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965aa76f_39b8_4f84_a5fc_08e784a6fee7.slice/crio-93c3e96a1c38941b2dc08cff2a447d69d2cbece3dd236514f025e465a6d66d4e WatchSource:0}: Error finding container 93c3e96a1c38941b2dc08cff2a447d69d2cbece3dd236514f025e465a6d66d4e: Status 404 returned error can't find the container with id 93c3e96a1c38941b2dc08cff2a447d69d2cbece3dd236514f025e465a6d66d4e
Sep 11 14:15:00.596559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.596539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"]
Sep 11 14:15:00.604866 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.604825    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"]
Sep 11 14:15:00.606854 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:15:00.606822    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76b5bd0_2014_447c_b333_2dc17d291bf2.slice/crio-9eeb24c52b32e8e1bdcc6b88259334a8d4041db7b6cbaa3ccfa8ab9fbbd91bae WatchSource:0}: Error finding container 9eeb24c52b32e8e1bdcc6b88259334a8d4041db7b6cbaa3ccfa8ab9fbbd91bae: Status 404 returned error can't find the container with id 9eeb24c52b32e8e1bdcc6b88259334a8d4041db7b6cbaa3ccfa8ab9fbbd91bae
Sep 11 14:15:00.693987 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.693952    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq" event={"ID":"965aa76f-39b8-4f84-a5fc-08e784a6fee7","Type":"ContainerStarted","Data":"93c3e96a1c38941b2dc08cff2a447d69d2cbece3dd236514f025e465a6d66d4e"}
Sep 11 14:15:00.695190 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:00.695170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4" event={"ID":"b76b5bd0-2014-447c-b333-2dc17d291bf2","Type":"ContainerStarted","Data":"9eeb24c52b32e8e1bdcc6b88259334a8d4041db7b6cbaa3ccfa8ab9fbbd91bae"}
Sep 11 14:15:01.702001 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:01.701885    2561 generic.go:356] "Generic (PLEG): container finished" podID="965aa76f-39b8-4f84-a5fc-08e784a6fee7" containerID="b694821d851d63a15d7fa41d00eee80322bc761d07b1b135d5ebe8c469648e98" exitCode=0
Sep 11 14:15:01.702001 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:01.701968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq" event={"ID":"965aa76f-39b8-4f84-a5fc-08e784a6fee7","Type":"ContainerDied","Data":"b694821d851d63a15d7fa41d00eee80322bc761d07b1b135d5ebe8c469648e98"}
Sep 11 14:15:01.703562 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:01.703533    2561 generic.go:356] "Generic (PLEG): container finished" podID="b76b5bd0-2014-447c-b333-2dc17d291bf2" containerID="55b4e4951f3416e5fef5d9c8661061016576845053fa87c4e4a5b36c2c72ffee" exitCode=0
Sep 11 14:15:01.703684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:01.703610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4" event={"ID":"b76b5bd0-2014-447c-b333-2dc17d291bf2","Type":"ContainerDied","Data":"55b4e4951f3416e5fef5d9c8661061016576845053fa87c4e4a5b36c2c72ffee"}
Sep 11 14:15:02.372047 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:02.372010    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"]
Sep 11 14:15:02.835478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:02.835452    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"
Sep 11 14:15:02.862602 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:02.862581    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"
Sep 11 14:15:03.714881 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:03.714850    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"
Sep 11 14:15:03.715105 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:03.714882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq" event={"ID":"965aa76f-39b8-4f84-a5fc-08e784a6fee7","Type":"ContainerDied","Data":"93c3e96a1c38941b2dc08cff2a447d69d2cbece3dd236514f025e465a6d66d4e"}
Sep 11 14:15:03.715105 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:03.714959    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c3e96a1c38941b2dc08cff2a447d69d2cbece3dd236514f025e465a6d66d4e"
Sep 11 14:15:03.716257 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:03.716239    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4"
Sep 11 14:15:03.716371 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:03.716255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-qkrl4" event={"ID":"b76b5bd0-2014-447c-b333-2dc17d291bf2","Type":"ContainerDied","Data":"9eeb24c52b32e8e1bdcc6b88259334a8d4041db7b6cbaa3ccfa8ab9fbbd91bae"}
Sep 11 14:15:03.716371 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:03.716287    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eeb24c52b32e8e1bdcc6b88259334a8d4041db7b6cbaa3ccfa8ab9fbbd91bae"
Sep 11 14:15:09.694110 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:09.694075    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:15:17.390787 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.390745    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255" podUID="20ee64a0-ca84-40f1-9b42-9f91369bf10b" containerName="main" containerID="cri-o://df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070" gracePeriod=45
Sep 11 14:15:17.634283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.634254    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:15:17.704059 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.703986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-model-cache\") pod \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") "
Sep 11 14:15:17.704059 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tls-certs\") pod \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") "
Sep 11 14:15:17.704274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-dshm\") pod \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") "
Sep 11 14:15:17.704274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tmp-dir\") pod \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") "
Sep 11 14:15:17.704274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-home\") pod \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") "
Sep 11 14:15:17.704274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704144    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgwkt\" (UniqueName: \"kubernetes.io/projected/20ee64a0-ca84-40f1-9b42-9f91369bf10b-kube-api-access-hgwkt\") pod \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\" (UID: \"20ee64a0-ca84-40f1-9b42-9f91369bf10b\") "
Sep 11 14:15:17.704274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-model-cache" (OuterVolumeSpecName: "model-cache") pod "20ee64a0-ca84-40f1-9b42-9f91369bf10b" (UID: "20ee64a0-ca84-40f1-9b42-9f91369bf10b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:15:17.704545 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20ee64a0-ca84-40f1-9b42-9f91369bf10b" (UID: "20ee64a0-ca84-40f1-9b42-9f91369bf10b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:15:17.704545 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704417    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:17.704630 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.704544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-home" (OuterVolumeSpecName: "home") pod "20ee64a0-ca84-40f1-9b42-9f91369bf10b" (UID: "20ee64a0-ca84-40f1-9b42-9f91369bf10b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:15:17.706416 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.706392    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20ee64a0-ca84-40f1-9b42-9f91369bf10b" (UID: "20ee64a0-ca84-40f1-9b42-9f91369bf10b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:15:17.706608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.706589    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-dshm" (OuterVolumeSpecName: "dshm") pod "20ee64a0-ca84-40f1-9b42-9f91369bf10b" (UID: "20ee64a0-ca84-40f1-9b42-9f91369bf10b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:15:17.706713 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.706689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ee64a0-ca84-40f1-9b42-9f91369bf10b-kube-api-access-hgwkt" (OuterVolumeSpecName: "kube-api-access-hgwkt") pod "20ee64a0-ca84-40f1-9b42-9f91369bf10b" (UID: "20ee64a0-ca84-40f1-9b42-9f91369bf10b"). InnerVolumeSpecName "kube-api-access-hgwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:15:17.785086 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.785053    2561 generic.go:356] "Generic (PLEG): container finished" podID="20ee64a0-ca84-40f1-9b42-9f91369bf10b" containerID="df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070" exitCode=0
Sep 11 14:15:17.785219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.785104    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"
Sep 11 14:15:17.785219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.785126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255" event={"ID":"20ee64a0-ca84-40f1-9b42-9f91369bf10b","Type":"ContainerDied","Data":"df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070"}
Sep 11 14:15:17.785219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.785169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255" event={"ID":"20ee64a0-ca84-40f1-9b42-9f91369bf10b","Type":"ContainerDied","Data":"44bcbba689852604b64de9b4ccd14c9abe60f0ae05ffbd6b48adf412d42e70a9"}
Sep 11 14:15:17.785219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.785199    2561 scope.go:117] "RemoveContainer" containerID="df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070"
Sep 11 14:15:17.794467 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.794447    2561 scope.go:117] "RemoveContainer" containerID="df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070"
Sep 11 14:15:17.794739 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:15:17.794720    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070\": container with ID starting with df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070 not found: ID does not exist" containerID="df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070"
Sep 11 14:15:17.794776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.794749    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070"} err="failed to get container status \"df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070\": rpc error: code = NotFound desc = could not find container \"df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070\": container with ID starting with df2d7e98a690378d661dcea54648bce28ed7022a1cdb88ed3ac8d68459168070 not found: ID does not exist"
Sep 11 14:15:17.805439 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.805414    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:17.805439 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.805437    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgwkt\" (UniqueName: \"kubernetes.io/projected/20ee64a0-ca84-40f1-9b42-9f91369bf10b-kube-api-access-hgwkt\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:17.805566 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.805447    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:17.805566 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.805455    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:17.805566 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.805463    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee64a0-ca84-40f1-9b42-9f91369bf10b-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:17.811010 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.810991    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"]
Sep 11 14:15:17.817368 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:17.817342    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d630fb00/e2e-lifecycle-model-d630fb00-kserve-5f7c48f887-gx255"]
Sep 11 14:15:18.121892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:18.121848    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ee64a0-ca84-40f1-9b42-9f91369bf10b" path="/var/lib/kubelet/pods/20ee64a0-ca84-40f1-9b42-9f91369bf10b/volumes"
Sep 11 14:15:24.006282 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:24.006247    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:15:32.269360 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:32.269323    2561 scope.go:117] "RemoveContainer" containerID="a98708d7cd1eb4ad3d17881b0c47a78335c79ccd639ebea622b97299114bab41"
Sep 11 14:15:40.202217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:40.202181    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:15:48.727401 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727367    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-0f7419"]
Sep 11 14:15:48.727865 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727820    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ee64a0-ca84-40f1-9b42-9f91369bf10b" containerName="main"
Sep 11 14:15:48.727865 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ee64a0-ca84-40f1-9b42-9f91369bf10b" containerName="main"
Sep 11 14:15:48.727865 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965aa76f-39b8-4f84-a5fc-08e784a6fee7" containerName="cleanup"
Sep 11 14:15:48.727865 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727861    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="965aa76f-39b8-4f84-a5fc-08e784a6fee7" containerName="cleanup"
Sep 11 14:15:48.728109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727880    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76b5bd0-2014-447c-b333-2dc17d291bf2" containerName="cleanup"
Sep 11 14:15:48.728109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727888    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76b5bd0-2014-447c-b333-2dc17d291bf2" containerName="cleanup"
Sep 11 14:15:48.728109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.727996    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ee64a0-ca84-40f1-9b42-9f91369bf10b" containerName="main"
Sep 11 14:15:48.728109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.728009    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76b5bd0-2014-447c-b333-2dc17d291bf2" containerName="cleanup"
Sep 11 14:15:48.728109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.728019    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="965aa76f-39b8-4f84-a5fc-08e784a6fee7" containerName="cleanup"
Sep 11 14:15:48.732397 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.732376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:48.735802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.735779    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m8d75\""
Sep 11 14:15:48.738717 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.738693    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-0f7419"]
Sep 11 14:15:48.775243 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.775192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ss25\" (UniqueName: \"kubernetes.io/projected/7df5024e-41fa-4109-a5c9-00589c3da6d7-kube-api-access-2ss25\") pod \"test-curl-7749-0f7419\" (UID: \"7df5024e-41fa-4109-a5c9-00589c3da6d7\") " pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:48.876550 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.876497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ss25\" (UniqueName: \"kubernetes.io/projected/7df5024e-41fa-4109-a5c9-00589c3da6d7-kube-api-access-2ss25\") pod \"test-curl-7749-0f7419\" (UID: \"7df5024e-41fa-4109-a5c9-00589c3da6d7\") " pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:48.888572 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:48.888529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ss25\" (UniqueName: \"kubernetes.io/projected/7df5024e-41fa-4109-a5c9-00589c3da6d7-kube-api-access-2ss25\") pod \"test-curl-7749-0f7419\" (UID: \"7df5024e-41fa-4109-a5c9-00589c3da6d7\") " pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:49.042760 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:49.042664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:49.313073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:49.313047    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-0f7419"]
Sep 11 14:15:49.315680 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:15:49.315654    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7df5024e_41fa_4109_a5c9_00589c3da6d7.slice/crio-162660c4da07a56afb003b16bf3339e5665256c0d8a3174c11b9c3a646ddea3f WatchSource:0}: Error finding container 162660c4da07a56afb003b16bf3339e5665256c0d8a3174c11b9c3a646ddea3f: Status 404 returned error can't find the container with id 162660c4da07a56afb003b16bf3339e5665256c0d8a3174c11b9c3a646ddea3f
Sep 11 14:15:49.936107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:49.936071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-0f7419" event={"ID":"7df5024e-41fa-4109-a5c9-00589c3da6d7","Type":"ContainerStarted","Data":"9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40"}
Sep 11 14:15:49.936107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:49.936114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-0f7419" event={"ID":"7df5024e-41fa-4109-a5c9-00589c3da6d7","Type":"ContainerStarted","Data":"162660c4da07a56afb003b16bf3339e5665256c0d8a3174c11b9c3a646ddea3f"}
Sep 11 14:15:50.168134 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:50.168082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7749-0f7419" podStartSLOduration=2.16806683 podStartE2EDuration="2.16806683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:49.955598864 +0000 UTC m=+1398.323188308" watchObservedRunningTime="2026-09-11 14:15:50.16806683 +0000 UTC m=+1398.535656273"
Sep 11 14:15:50.652414 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:50.652332    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-0f7419"]
Sep 11 14:15:50.664583 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:50.664547    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-0f7419"]
Sep 11 14:15:50.940950 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:15:50.940849    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7749-0f7419" container="test-curl-7749-0f7419"
Sep 11 14:15:51.946236 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:51.946172    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7749-0f7419" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" containerName="test-curl-7749-0f7419" containerID="cri-o://9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40" gracePeriod=2
Sep 11 14:15:51.959849 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:15:51.959808    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-0f7419\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:52.115970 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:15:52.115927    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-0f7419\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:54.122139 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.122115    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:54.218766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.218686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ss25\" (UniqueName: \"kubernetes.io/projected/7df5024e-41fa-4109-a5c9-00589c3da6d7-kube-api-access-2ss25\") pod \"7df5024e-41fa-4109-a5c9-00589c3da6d7\" (UID: \"7df5024e-41fa-4109-a5c9-00589c3da6d7\") "
Sep 11 14:15:54.220792 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.220764    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df5024e-41fa-4109-a5c9-00589c3da6d7-kube-api-access-2ss25" (OuterVolumeSpecName: "kube-api-access-2ss25") pod "7df5024e-41fa-4109-a5c9-00589c3da6d7" (UID: "7df5024e-41fa-4109-a5c9-00589c3da6d7"). InnerVolumeSpecName "kube-api-access-2ss25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:15:54.319764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.319731    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ss25\" (UniqueName: \"kubernetes.io/projected/7df5024e-41fa-4109-a5c9-00589c3da6d7-kube-api-access-2ss25\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:15:54.741134 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.741099    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:15:54.961876 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.961840    2561 generic.go:356] "Generic (PLEG): container finished" podID="7df5024e-41fa-4109-a5c9-00589c3da6d7" containerID="9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40" exitCode=137
Sep 11 14:15:54.962046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.961889    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:54.962046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.961964    2561 scope.go:117] "RemoveContainer" containerID="9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40"
Sep 11 14:15:54.970605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.970584    2561 scope.go:117] "RemoveContainer" containerID="9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40"
Sep 11 14:15:54.970892 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:15:54.970872    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40\": container with ID starting with 9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40 not found: ID does not exist" containerID="9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40"
Sep 11 14:15:54.970960 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:54.970901    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40"} err="failed to get container status \"9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40\": rpc error: code = NotFound desc = could not find container \"9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40\": container with ID starting with 9b96ded6ef5fe9a50584748b0bf701badf57a495db18b9c7bc29adb80564fe40 not found: ID does not exist"
Sep 11 14:15:54.972565 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:15:54.972544    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-0f7419\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" pod="opendatahub/test-curl-7749-0f7419"
Sep 11 14:15:56.116372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:15:56.116339    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" path="/var/lib/kubelet/pods/7df5024e-41fa-4109-a5c9-00589c3da6d7/volumes"
Sep 11 14:16:12.100719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:12.100683    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"]
Sep 11 14:16:13.326864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:13.326826    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"]
Sep 11 14:16:16.603621 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:16.603578    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:16:27.120962 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.120833    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675" podUID="80447a04-cb48-42c7-931b-fde90370f062" containerName="main" containerID="cri-o://3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7" gracePeriod=45
Sep 11 14:16:27.370905 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.370879    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:16:27.522449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-dshm\") pod \"80447a04-cb48-42c7-931b-fde90370f062\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") "
Sep 11 14:16:27.522449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-tmp-dir\") pod \"80447a04-cb48-42c7-931b-fde90370f062\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") "
Sep 11 14:16:27.522449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-home\") pod \"80447a04-cb48-42c7-931b-fde90370f062\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") "
Sep 11 14:16:27.522735 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-model-cache\") pod \"80447a04-cb48-42c7-931b-fde90370f062\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") "
Sep 11 14:16:27.522735 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522526    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80447a04-cb48-42c7-931b-fde90370f062-tls-certs\") pod \"80447a04-cb48-42c7-931b-fde90370f062\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") "
Sep 11 14:16:27.522735 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522574    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp88t\" (UniqueName: \"kubernetes.io/projected/80447a04-cb48-42c7-931b-fde90370f062-kube-api-access-rp88t\") pod \"80447a04-cb48-42c7-931b-fde90370f062\" (UID: \"80447a04-cb48-42c7-931b-fde90370f062\") "
Sep 11 14:16:27.522897 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522766    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80447a04-cb48-42c7-931b-fde90370f062" (UID: "80447a04-cb48-42c7-931b-fde90370f062"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:27.522897 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-home" (OuterVolumeSpecName: "home") pod "80447a04-cb48-42c7-931b-fde90370f062" (UID: "80447a04-cb48-42c7-931b-fde90370f062"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:27.522897 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.522849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-model-cache" (OuterVolumeSpecName: "model-cache") pod "80447a04-cb48-42c7-931b-fde90370f062" (UID: "80447a04-cb48-42c7-931b-fde90370f062"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:27.524814 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.524790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-dshm" (OuterVolumeSpecName: "dshm") pod "80447a04-cb48-42c7-931b-fde90370f062" (UID: "80447a04-cb48-42c7-931b-fde90370f062"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:27.524950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.524831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80447a04-cb48-42c7-931b-fde90370f062-kube-api-access-rp88t" (OuterVolumeSpecName: "kube-api-access-rp88t") pod "80447a04-cb48-42c7-931b-fde90370f062" (UID: "80447a04-cb48-42c7-931b-fde90370f062"). InnerVolumeSpecName "kube-api-access-rp88t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:16:27.524950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.524853    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80447a04-cb48-42c7-931b-fde90370f062-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80447a04-cb48-42c7-931b-fde90370f062" (UID: "80447a04-cb48-42c7-931b-fde90370f062"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:16:27.623332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.623295    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:27.623332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.623325    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:27.623332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.623335    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:27.623332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.623343    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80447a04-cb48-42c7-931b-fde90370f062-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:27.623606 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.623350    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80447a04-cb48-42c7-931b-fde90370f062-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:27.623606 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:27.623359    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp88t\" (UniqueName: \"kubernetes.io/projected/80447a04-cb48-42c7-931b-fde90370f062-kube-api-access-rp88t\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:28.110078 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.110036    2561 generic.go:356] "Generic (PLEG): container finished" podID="80447a04-cb48-42c7-931b-fde90370f062" containerID="3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7" exitCode=0
Sep 11 14:16:28.110264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.110093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675" event={"ID":"80447a04-cb48-42c7-931b-fde90370f062","Type":"ContainerDied","Data":"3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7"}
Sep 11 14:16:28.110264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.110099    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"
Sep 11 14:16:28.110264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.110132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675" event={"ID":"80447a04-cb48-42c7-931b-fde90370f062","Type":"ContainerDied","Data":"ab5b7c28d4cdc3d5a239d6629baa3cc20062ea41377be8bb2e3a1602a59b9e11"}
Sep 11 14:16:28.110264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.110151    2561 scope.go:117] "RemoveContainer" containerID="3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7"
Sep 11 14:16:28.119764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.119728    2561 scope.go:117] "RemoveContainer" containerID="3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7"
Sep 11 14:16:28.120103 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:16:28.120078    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7\": container with ID starting with 3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7 not found: ID does not exist" containerID="3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7"
Sep 11 14:16:28.120217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.120114    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7"} err="failed to get container status \"3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7\": rpc error: code = NotFound desc = could not find container \"3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7\": container with ID starting with 3ddf22bd8dd3bd68db38bdfaf885208b3185a46d4e6b4bdc1970debb1eb7a9f7 not found: ID does not exist"
Sep 11 14:16:28.345365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.345305    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" podUID="1960bc53-76b1-4012-95ca-37d7e7361ffa" containerName="main" containerID="cri-o://5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372" gracePeriod=45
Sep 11 14:16:28.612941 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.612898    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:16:28.734019 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.733903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-model-cache\") pod \"1960bc53-76b1-4012-95ca-37d7e7361ffa\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") "
Sep 11 14:16:28.734019 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-tmp-dir\") pod \"1960bc53-76b1-4012-95ca-37d7e7361ffa\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") "
Sep 11 14:16:28.734237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-home\") pod \"1960bc53-76b1-4012-95ca-37d7e7361ffa\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") "
Sep 11 14:16:28.734237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-dshm\") pod \"1960bc53-76b1-4012-95ca-37d7e7361ffa\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") "
Sep 11 14:16:28.734237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs\") pod \"1960bc53-76b1-4012-95ca-37d7e7361ffa\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") "
Sep 11 14:16:28.734237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734144    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfrqs\" (UniqueName: \"kubernetes.io/projected/1960bc53-76b1-4012-95ca-37d7e7361ffa-kube-api-access-vfrqs\") pod \"1960bc53-76b1-4012-95ca-37d7e7361ffa\" (UID: \"1960bc53-76b1-4012-95ca-37d7e7361ffa\") "
Sep 11 14:16:28.734237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734189    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-model-cache" (OuterVolumeSpecName: "model-cache") pod "1960bc53-76b1-4012-95ca-37d7e7361ffa" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:28.734497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-home" (OuterVolumeSpecName: "home") pod "1960bc53-76b1-4012-95ca-37d7e7361ffa" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:28.734497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1960bc53-76b1-4012-95ca-37d7e7361ffa" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:28.734497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734436    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:28.734497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734454    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:28.734497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.734464    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:28.736387 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.736354    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-dshm" (OuterVolumeSpecName: "dshm") pod "1960bc53-76b1-4012-95ca-37d7e7361ffa" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:16:28.736501 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.736405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1960bc53-76b1-4012-95ca-37d7e7361ffa" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:16:28.736501 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.736471    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1960bc53-76b1-4012-95ca-37d7e7361ffa-kube-api-access-vfrqs" (OuterVolumeSpecName: "kube-api-access-vfrqs") pod "1960bc53-76b1-4012-95ca-37d7e7361ffa" (UID: "1960bc53-76b1-4012-95ca-37d7e7361ffa"). InnerVolumeSpecName "kube-api-access-vfrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:16:28.834917 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.834888    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1960bc53-76b1-4012-95ca-37d7e7361ffa-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:28.835087 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.834924    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1960bc53-76b1-4012-95ca-37d7e7361ffa-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:28.835087 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:28.834939    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfrqs\" (UniqueName: \"kubernetes.io/projected/1960bc53-76b1-4012-95ca-37d7e7361ffa-kube-api-access-vfrqs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:16:29.117553 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.117513    2561 generic.go:356] "Generic (PLEG): container finished" podID="1960bc53-76b1-4012-95ca-37d7e7361ffa" containerID="5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372" exitCode=0
Sep 11 14:16:29.117708 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.117559    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"
Sep 11 14:16:29.117708 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.117590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" event={"ID":"1960bc53-76b1-4012-95ca-37d7e7361ffa","Type":"ContainerDied","Data":"5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372"}
Sep 11 14:16:29.117708 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.117634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp" event={"ID":"1960bc53-76b1-4012-95ca-37d7e7361ffa","Type":"ContainerDied","Data":"71459cdd67560d0e79e81366ba6af18e78fe1c02eaa9b4187d2725a074f0f970"}
Sep 11 14:16:29.117708 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.117653    2561 scope.go:117] "RemoveContainer" containerID="5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372"
Sep 11 14:16:29.126891 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.126875    2561 scope.go:117] "RemoveContainer" containerID="5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372"
Sep 11 14:16:29.127219 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:16:29.127202    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372\": container with ID starting with 5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372 not found: ID does not exist" containerID="5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372"
Sep 11 14:16:29.127281 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:29.127229    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372"} err="failed to get container status \"5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372\": rpc error: code = NotFound desc = could not find container \"5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372\": container with ID starting with 5245cc4ab22d7088ff78ccd1a11c98e9dcb3967a121336234cc3d339437fd372 not found: ID does not exist"
Sep 11 14:16:32.756119 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:32.756088    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"]
Sep 11 14:16:33.994716 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.994681    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"]
Sep 11 14:16:33.995118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995028    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1960bc53-76b1-4012-95ca-37d7e7361ffa" containerName="main"
Sep 11 14:16:33.995118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995040    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1960bc53-76b1-4012-95ca-37d7e7361ffa" containerName="main"
Sep 11 14:16:33.995118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995051    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" containerName="test-curl-7749-0f7419"
Sep 11 14:16:33.995118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995057    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" containerName="test-curl-7749-0f7419"
Sep 11 14:16:33.995118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995070    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80447a04-cb48-42c7-931b-fde90370f062" containerName="main"
Sep 11 14:16:33.995118 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995076    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="80447a04-cb48-42c7-931b-fde90370f062" containerName="main"
Sep 11 14:16:33.995329 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995129    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1960bc53-76b1-4012-95ca-37d7e7361ffa" containerName="main"
Sep 11 14:16:33.995329 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995139    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="80447a04-cb48-42c7-931b-fde90370f062" containerName="main"
Sep 11 14:16:33.995329 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.995147    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df5024e-41fa-4109-a5c9-00589c3da6d7" containerName="test-curl-7749-0f7419"
Sep 11 14:16:33.999357 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:33.999340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.003294 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.003272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"sub-test-model-fb65d4-kserve-self-signed-certs\""
Sep 11 14:16:34.004296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.004278    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"kube-root-ca.crt\""
Sep 11 14:16:34.004687 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.004481    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"openshift-service-ca.crt\""
Sep 11 14:16:34.004687 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.004535    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"default-dockercfg-gchl6\""
Sep 11 14:16:34.078887 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.078852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-dshm\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.079103 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.078934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tmp-dir\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.079103 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.078967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t28f\" (UniqueName: \"kubernetes.io/projected/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-kube-api-access-7t28f\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.079103 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.078985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tls-certs\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.079103 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.079011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-home\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.079245 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.079102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-model-cache\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.179512 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.179469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-dshm\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.179712 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.179539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tmp-dir\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.179712 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.179568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t28f\" (UniqueName: \"kubernetes.io/projected/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-kube-api-access-7t28f\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.179712 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.179592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tls-certs\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.179712 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.179628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-home\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.179991 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.179771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-model-cache\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.180055 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.180024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tmp-dir\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.180055 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.180032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-home\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.180142 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.180130    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-model-cache\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.181874 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.181857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-dshm\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:34.182072 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:34.182055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tls-certs\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:35.471356 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:35.471320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"]
Sep 11 14:16:35.472942 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:35.472897    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/auth-test-model-fb65d4-kserve-79586b6c6c-p6675"]
Sep 11 14:16:35.488655 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:35.488631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t28f\" (UniqueName: \"kubernetes.io/projected/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-kube-api-access-7t28f\") pod \"sub-test-model-fb65d4-kserve-65cd895f89-4mb2p\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:35.508546 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:35.508523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:36.117105 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:36.117074    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80447a04-cb48-42c7-931b-fde90370f062" path="/var/lib/kubelet/pods/80447a04-cb48-42c7-931b-fde90370f062/volumes"
Sep 11 14:16:37.532535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:37.532473    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"]
Sep 11 14:16:38.162211 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:38.162161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p" event={"ID":"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41","Type":"ContainerStarted","Data":"2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93"}
Sep 11 14:16:38.162211 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:38.162217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p" event={"ID":"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41","Type":"ContainerStarted","Data":"8420350e981534be18771bdc608c569c261b223ebbbaabe79024a89c8496c9e2"}
Sep 11 14:16:38.676644 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:38.676607    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"]
Sep 11 14:16:39.470212 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:39.470174    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/auth-test-model-261f9c-kserve-5b776fdc4c-lkkrp"]
Sep 11 14:16:40.117448 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:40.117410    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1960bc53-76b1-4012-95ca-37d7e7361ffa" path="/var/lib/kubelet/pods/1960bc53-76b1-4012-95ca-37d7e7361ffa/volumes"
Sep 11 14:16:41.891174 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:41.891107    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p" podStartSLOduration=11.891089973 podStartE2EDuration="11.891089973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:41.888350423 +0000 UTC m=+1450.255939865" watchObservedRunningTime="2026-09-11 14:16:41.891089973 +0000 UTC m=+1450.258679418"
Sep 11 14:16:45.509533 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:45.509499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:45.509992 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:45.509780    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:45.522442 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:45.522419    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:46.211441 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:46.211408    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:16:53.912374 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:16:53.912332    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:17:14.915079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.915047    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"]
Sep 11 14:17:14.918446 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.918430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:14.921345 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.921324    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"kube-root-ca.crt\""
Sep 11 14:17:14.922610 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.922592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"openshift-service-ca.crt\""
Sep 11 14:17:14.922610 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.922606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"default-dockercfg-qjcj7\""
Sep 11 14:17:14.922784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.922719    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"sub-test-model-261f9c-kserve-self-signed-certs\""
Sep 11 14:17:14.934975 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:14.934950    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"]
Sep 11 14:17:15.026703 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.026669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-dshm\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.026703 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.026707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-model-cache\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.026979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.026729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tls-certs\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.026979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.026814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpj2\" (UniqueName: \"kubernetes.io/projected/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-kube-api-access-bcpj2\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.026979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.026855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-home\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.026979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.026877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tmp-dir\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128226 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tmp-dir\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128423 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-dshm\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128423 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-model-cache\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128423 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tls-certs\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128423 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpj2\" (UniqueName: \"kubernetes.io/projected/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-kube-api-access-bcpj2\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128423 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-home\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128685 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tmp-dir\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128755 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-model-cache\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.128858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.128839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-home\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.130673 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.130654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-dshm\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.130827 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.130810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tls-certs\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.143069 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.143043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpj2\" (UniqueName: \"kubernetes.io/projected/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-kube-api-access-bcpj2\") pod \"sub-test-model-261f9c-kserve-784786bf4f-9tjks\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.226994 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.226930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:15.504503 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:15.504429    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"]
Sep 11 14:17:15.507294 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:17:15.507272    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e2c484d_d7d7_49a4_85e7_b5936c3f9cea.slice/crio-ef198d5d171da796035a6167464110228e9eb725ad509a5cdd19496e054d551d WatchSource:0}: Error finding container ef198d5d171da796035a6167464110228e9eb725ad509a5cdd19496e054d551d: Status 404 returned error can't find the container with id ef198d5d171da796035a6167464110228e9eb725ad509a5cdd19496e054d551d
Sep 11 14:17:16.333744 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:16.333703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks" event={"ID":"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea","Type":"ContainerStarted","Data":"f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b"}
Sep 11 14:17:16.334200 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:16.333754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks" event={"ID":"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea","Type":"ContainerStarted","Data":"ef198d5d171da796035a6167464110228e9eb725ad509a5cdd19496e054d551d"}
Sep 11 14:17:16.393850 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:16.393792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks" podStartSLOduration=2.393776103 podStartE2EDuration="2.393776103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:16.393294206 +0000 UTC m=+1484.760883648" watchObservedRunningTime="2026-09-11 14:17:16.393776103 +0000 UTC m=+1484.761365546"
Sep 11 14:17:18.120397 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:18.120355    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:17:25.227946 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:25.227888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:25.228378 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:25.227966    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:25.240449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:25.240421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:25.386530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:25.386500    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:17:32.176556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:32.176528    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:17:32.180536 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:32.180513    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:17:32.311141 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:32.311110    2561 scope.go:117] "RemoveContainer" containerID="2c871bbb8729435e38b72f720351f459fdd8de24978f2891c1ff603cc18775c4"
Sep 11 14:17:32.318943 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:32.318924    2561 scope.go:117] "RemoveContainer" containerID="b8a03de7c5e2237139a875bf001c7719fb1926e675b4ebc7c6619d68ee849402"
Sep 11 14:17:39.901351 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:39.901315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:17:51.035325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.035285    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"]
Sep 11 14:17:51.038802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.038785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.044853 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.044832    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-17d457\""
Sep 11 14:17:51.045572 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.045554    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-17d457-dockercfg-8lr8l\""
Sep 11 14:17:51.057628 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.057599    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"]
Sep 11 14:17:51.151149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.151114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38be31a7-425b-4ed5-a4ba-e80331d8d64f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.151319 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.151162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbwz\" (UniqueName: \"kubernetes.io/projected/38be31a7-425b-4ed5-a4ba-e80331d8d64f-kube-api-access-cvbwz\") pod \"payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.251903 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.251863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38be31a7-425b-4ed5-a4ba-e80331d8d64f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.252097 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.251951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbwz\" (UniqueName: \"kubernetes.io/projected/38be31a7-425b-4ed5-a4ba-e80331d8d64f-kube-api-access-cvbwz\") pod \"payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.252547 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.252528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38be31a7-425b-4ed5-a4ba-e80331d8d64f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.261372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.261346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbwz\" (UniqueName: \"kubernetes.io/projected/38be31a7-425b-4ed5-a4ba-e80331d8d64f-kube-api-access-cvbwz\") pod \"payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.348558 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.348524    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:51.631342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:51.631254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"]
Sep 11 14:17:51.634296 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:17:51.634265    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38be31a7_425b_4ed5_a4ba_e80331d8d64f.slice/crio-d5decd7d0566f52ac72acb4c24c0ef010c9f9d730fc4abba8f36648dee3be8b3 WatchSource:0}: Error finding container d5decd7d0566f52ac72acb4c24c0ef010c9f9d730fc4abba8f36648dee3be8b3: Status 404 returned error can't find the container with id d5decd7d0566f52ac72acb4c24c0ef010c9f9d730fc4abba8f36648dee3be8b3
Sep 11 14:17:52.503613 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:52.503578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2" event={"ID":"38be31a7-425b-4ed5-a4ba-e80331d8d64f","Type":"ContainerStarted","Data":"b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543"}
Sep 11 14:17:52.504054 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:52.503627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2" event={"ID":"38be31a7-425b-4ed5-a4ba-e80331d8d64f","Type":"ContainerStarted","Data":"d5decd7d0566f52ac72acb4c24c0ef010c9f9d730fc4abba8f36648dee3be8b3"}
Sep 11 14:17:52.504054 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:52.503694    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:17:52.533086 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:52.533023    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2" podStartSLOduration=1.5330012 podStartE2EDuration="1.5330012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:52.531557136 +0000 UTC m=+1520.899146585" watchObservedRunningTime="2026-09-11 14:17:52.5330012 +0000 UTC m=+1520.900590648"
Sep 11 14:17:58.408827 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:17:58.408739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:18:00.958343 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:00.958302    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-7eea1f"]
Sep 11 14:18:00.963606 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:00.963573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:00.969184 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:00.969150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m8d75\""
Sep 11 14:18:00.974237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:00.974211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-7eea1f"]
Sep 11 14:18:01.146033 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:01.145995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrn7z\" (UniqueName: \"kubernetes.io/projected/226e8617-bd4f-4a89-9877-980ea3b9a1ff-kube-api-access-hrn7z\") pod \"test-curl-7749-7eea1f\" (UID: \"226e8617-bd4f-4a89-9877-980ea3b9a1ff\") " pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:01.246972 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:01.246850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn7z\" (UniqueName: \"kubernetes.io/projected/226e8617-bd4f-4a89-9877-980ea3b9a1ff-kube-api-access-hrn7z\") pod \"test-curl-7749-7eea1f\" (UID: \"226e8617-bd4f-4a89-9877-980ea3b9a1ff\") " pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:01.268047 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:01.268019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrn7z\" (UniqueName: \"kubernetes.io/projected/226e8617-bd4f-4a89-9877-980ea3b9a1ff-kube-api-access-hrn7z\") pod \"test-curl-7749-7eea1f\" (UID: \"226e8617-bd4f-4a89-9877-980ea3b9a1ff\") " pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:01.271854 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:01.271832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:01.552388 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:01.552315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-7eea1f"]
Sep 11 14:18:01.561665 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:18:01.561636    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod226e8617_bd4f_4a89_9877_980ea3b9a1ff.slice/crio-a27fa9feda3b8dea08547f25e8bd04384b560b68cad0dc3c37c43d504dcc0a83 WatchSource:0}: Error finding container a27fa9feda3b8dea08547f25e8bd04384b560b68cad0dc3c37c43d504dcc0a83: Status 404 returned error can't find the container with id a27fa9feda3b8dea08547f25e8bd04384b560b68cad0dc3c37c43d504dcc0a83
Sep 11 14:18:02.552714 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:02.552678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-7eea1f" event={"ID":"226e8617-bd4f-4a89-9877-980ea3b9a1ff","Type":"ContainerStarted","Data":"9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b"}
Sep 11 14:18:02.552714 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:02.552721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-7eea1f" event={"ID":"226e8617-bd4f-4a89-9877-980ea3b9a1ff","Type":"ContainerStarted","Data":"a27fa9feda3b8dea08547f25e8bd04384b560b68cad0dc3c37c43d504dcc0a83"}
Sep 11 14:18:02.586176 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:02.586117    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7749-7eea1f" podStartSLOduration=2.586099548 podStartE2EDuration="2.586099548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:02.584928593 +0000 UTC m=+1530.952518027" watchObservedRunningTime="2026-09-11 14:18:02.586099548 +0000 UTC m=+1530.953688990"
Sep 11 14:18:03.366939 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.366837    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-7eea1f"]
Sep 11 14:18:03.371435 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.371401    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-7eea1f"]
Sep 11 14:18:03.499034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.499001    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-dd7bfb"]
Sep 11 14:18:03.499359 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.499347    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" containerName="test-curl-7749-7eea1f"
Sep 11 14:18:03.499403 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.499360    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" containerName="test-curl-7749-7eea1f"
Sep 11 14:18:03.499447 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.499416    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" containerName="test-curl-7749-7eea1f"
Sep 11 14:18:03.502306 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.502290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:03.509881 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.509860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:18:03.515369 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.515342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-dd7bfb"]
Sep 11 14:18:03.557814 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:03.557780    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7749-7eea1f" container="test-curl-7749-7eea1f"
Sep 11 14:18:03.669947 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.669815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5jl\" (UniqueName: \"kubernetes.io/projected/7a58d13b-bf39-40b8-a1d8-55d523b16eba-kube-api-access-ql5jl\") pod \"test-curl-7749-dd7bfb\" (UID: \"7a58d13b-bf39-40b8-a1d8-55d523b16eba\") " pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:03.771073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.771016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5jl\" (UniqueName: \"kubernetes.io/projected/7a58d13b-bf39-40b8-a1d8-55d523b16eba-kube-api-access-ql5jl\") pod \"test-curl-7749-dd7bfb\" (UID: \"7a58d13b-bf39-40b8-a1d8-55d523b16eba\") " pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:03.792752 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.792717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5jl\" (UniqueName: \"kubernetes.io/projected/7a58d13b-bf39-40b8-a1d8-55d523b16eba-kube-api-access-ql5jl\") pod \"test-curl-7749-dd7bfb\" (UID: \"7a58d13b-bf39-40b8-a1d8-55d523b16eba\") " pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:03.811642 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:03.811605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:04.096139 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:04.096101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-dd7bfb"]
Sep 11 14:18:04.099053 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:18:04.099025    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a58d13b_bf39_40b8_a1d8_55d523b16eba.slice/crio-c974d0fa28b4c90182fd341aedf6bdc67237d51d283a4e61979960bd4189bbd2 WatchSource:0}: Error finding container c974d0fa28b4c90182fd341aedf6bdc67237d51d283a4e61979960bd4189bbd2: Status 404 returned error can't find the container with id c974d0fa28b4c90182fd341aedf6bdc67237d51d283a4e61979960bd4189bbd2
Sep 11 14:18:04.563131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:04.563097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-dd7bfb" event={"ID":"7a58d13b-bf39-40b8-a1d8-55d523b16eba","Type":"ContainerStarted","Data":"9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e"}
Sep 11 14:18:04.563579 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:04.563141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-dd7bfb" event={"ID":"7a58d13b-bf39-40b8-a1d8-55d523b16eba","Type":"ContainerStarted","Data":"c974d0fa28b4c90182fd341aedf6bdc67237d51d283a4e61979960bd4189bbd2"}
Sep 11 14:18:04.563579 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:04.563164    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7749-7eea1f" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" containerName="test-curl-7749-7eea1f" containerID="cri-o://9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b" gracePeriod=2
Sep 11 14:18:04.592019 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:04.591965    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7749-dd7bfb" podStartSLOduration=1.591947529 podStartE2EDuration="1.591947529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:04.59138898 +0000 UTC m=+1532.958978424" watchObservedRunningTime="2026-09-11 14:18:04.591947529 +0000 UTC m=+1532.959536972"
Sep 11 14:18:05.283604 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:05.283559    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-dd7bfb"]
Sep 11 14:18:05.290719 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:05.290639    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-dd7bfb"]
Sep 11 14:18:05.567743 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:05.567707    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7749-dd7bfb" container="test-curl-7749-dd7bfb"
Sep 11 14:18:06.572934 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:06.572886    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7749-dd7bfb" podUID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" containerName="test-curl-7749-dd7bfb" containerID="cri-o://9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e" gracePeriod=2
Sep 11 14:18:06.751779 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:06.751752    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:06.794523 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:06.794491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrn7z\" (UniqueName: \"kubernetes.io/projected/226e8617-bd4f-4a89-9877-980ea3b9a1ff-kube-api-access-hrn7z\") pod \"226e8617-bd4f-4a89-9877-980ea3b9a1ff\" (UID: \"226e8617-bd4f-4a89-9877-980ea3b9a1ff\") "
Sep 11 14:18:06.796519 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:06.796489    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226e8617-bd4f-4a89-9877-980ea3b9a1ff-kube-api-access-hrn7z" (OuterVolumeSpecName: "kube-api-access-hrn7z") pod "226e8617-bd4f-4a89-9877-980ea3b9a1ff" (UID: "226e8617-bd4f-4a89-9877-980ea3b9a1ff"). InnerVolumeSpecName "kube-api-access-hrn7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:06.895360 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:06.895270    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrn7z\" (UniqueName: \"kubernetes.io/projected/226e8617-bd4f-4a89-9877-980ea3b9a1ff-kube-api-access-hrn7z\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:07.578376 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:07.578337    2561 generic.go:356] "Generic (PLEG): container finished" podID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" containerID="9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b" exitCode=137
Sep 11 14:18:07.578863 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:07.578386    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:07.578863 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:07.578404    2561 scope.go:117] "RemoveContainer" containerID="9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b"
Sep 11 14:18:07.586749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:07.586728    2561 scope.go:117] "RemoveContainer" containerID="9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b"
Sep 11 14:18:07.587067 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:07.587047    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b\": container with ID starting with 9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b not found: ID does not exist" containerID="9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b"
Sep 11 14:18:07.587155 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:07.587074    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b"} err="failed to get container status \"9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b\": rpc error: code = NotFound desc = could not find container \"9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b\": container with ID starting with 9b2c8e29f7102b7d287b4accca13e313abae042b200cbf62dc0620d299f3139b not found: ID does not exist"
Sep 11 14:18:07.590975 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:07.590943    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-7eea1f\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:07.709373 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:07.709321    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"]
Sep 11 14:18:07.753017 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:07.752976    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-7eea1f\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" pod="opendatahub/test-curl-7749-7eea1f"
Sep 11 14:18:08.116672 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:08.116636    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226e8617-bd4f-4a89-9877-980ea3b9a1ff" path="/var/lib/kubelet/pods/226e8617-bd4f-4a89-9877-980ea3b9a1ff/volumes"
Sep 11 14:18:08.697388 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:08.697354    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"]
Sep 11 14:18:08.762510 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:08.762480    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:08.810808 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:08.810777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql5jl\" (UniqueName: \"kubernetes.io/projected/7a58d13b-bf39-40b8-a1d8-55d523b16eba-kube-api-access-ql5jl\") pod \"7a58d13b-bf39-40b8-a1d8-55d523b16eba\" (UID: \"7a58d13b-bf39-40b8-a1d8-55d523b16eba\") "
Sep 11 14:18:08.812782 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:08.812759    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a58d13b-bf39-40b8-a1d8-55d523b16eba-kube-api-access-ql5jl" (OuterVolumeSpecName: "kube-api-access-ql5jl") pod "7a58d13b-bf39-40b8-a1d8-55d523b16eba" (UID: "7a58d13b-bf39-40b8-a1d8-55d523b16eba"). InnerVolumeSpecName "kube-api-access-ql5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:08.912071 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:08.911988    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5jl\" (UniqueName: \"kubernetes.io/projected/7a58d13b-bf39-40b8-a1d8-55d523b16eba-kube-api-access-ql5jl\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:09.589749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:09.589715    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" containerID="9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e" exitCode=137
Sep 11 14:18:09.589949 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:09.589758    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:09.589949 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:09.589783    2561 scope.go:117] "RemoveContainer" containerID="9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e"
Sep 11 14:18:09.598220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:09.598201    2561 scope.go:117] "RemoveContainer" containerID="9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e"
Sep 11 14:18:09.598506 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:09.598486    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e\": container with ID starting with 9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e not found: ID does not exist" containerID="9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e"
Sep 11 14:18:09.598577 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:09.598514    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e"} err="failed to get container status \"9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e\": rpc error: code = NotFound desc = could not find container \"9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e\": container with ID starting with 9d6b93d8bf557b806625bd403eff2c1d2a932bc07f8375f5cbf27c705105131e not found: ID does not exist"
Sep 11 14:18:09.600160 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:09.600137    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-dd7bfb\" is forbidden: User \"system:node:ip-10-0-128-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-105.ec2.internal' and this object" podUID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" pod="opendatahub/test-curl-7749-dd7bfb"
Sep 11 14:18:10.116495 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:10.116465    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" path="/var/lib/kubelet/pods/7a58d13b-bf39-40b8-a1d8-55d523b16eba/volumes"
Sep 11 14:18:13.006028 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.005981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:18:13.295122 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.295038    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"]
Sep 11 14:18:13.295369 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.295358    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" containerName="test-curl-7749-dd7bfb"
Sep 11 14:18:13.295418 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.295372    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" containerName="test-curl-7749-dd7bfb"
Sep 11 14:18:13.295454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.295436    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a58d13b-bf39-40b8-a1d8-55d523b16eba" containerName="test-curl-7749-dd7bfb"
Sep 11 14:18:13.299605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.299586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.303104 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.303075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fb65d4\"/\"rate-test-model-fb65d4-kserve-self-signed-certs\""
Sep 11 14:18:13.309859 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.309835    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"]
Sep 11 14:18:13.338773 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.338738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-model-cache\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.338981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.338780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-home\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.338981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.338839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-tmp-dir\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.338981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.338870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1efd8b-e438-4e28-b48f-73260c652462-tls-certs\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.338981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.338894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqnr\" (UniqueName: \"kubernetes.io/projected/1b1efd8b-e438-4e28-b48f-73260c652462-kube-api-access-xgqnr\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.338981 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.338971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-dshm\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439361 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-home\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-tmp-dir\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1efd8b-e438-4e28-b48f-73260c652462-tls-certs\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqnr\" (UniqueName: \"kubernetes.io/projected/1b1efd8b-e438-4e28-b48f-73260c652462-kube-api-access-xgqnr\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-dshm\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-model-cache\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.439888 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-tmp-dir\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.440025 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.439886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-home\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.440089 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.440028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-model-cache\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.442191 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.442167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-dshm\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.442293 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.442271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1efd8b-e438-4e28-b48f-73260c652462-tls-certs\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.459121 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.459087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqnr\" (UniqueName: \"kubernetes.io/projected/1b1efd8b-e438-4e28-b48f-73260c652462-kube-api-access-xgqnr\") pod \"rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") " pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.609593 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.609563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:13.903202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:13.903175    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"]
Sep 11 14:18:13.905261 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:18:13.905234    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b1efd8b_e438_4e28_b48f_73260c652462.slice/crio-b27883b517004d8030b0999fe60445cb34cb175ec14e9920a5ebc53b6ec3369e WatchSource:0}: Error finding container b27883b517004d8030b0999fe60445cb34cb175ec14e9920a5ebc53b6ec3369e: Status 404 returned error can't find the container with id b27883b517004d8030b0999fe60445cb34cb175ec14e9920a5ebc53b6ec3369e
Sep 11 14:18:14.614364 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:14.614325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh" event={"ID":"1b1efd8b-e438-4e28-b48f-73260c652462","Type":"ContainerStarted","Data":"c8a150123fa6e2701554a8a3c828805ae4f66cc38477adddb4ce75a43889799c"}
Sep 11 14:18:14.614758 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:14.614377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh" event={"ID":"1b1efd8b-e438-4e28-b48f-73260c652462","Type":"ContainerStarted","Data":"b27883b517004d8030b0999fe60445cb34cb175ec14e9920a5ebc53b6ec3369e"}
Sep 11 14:18:14.635857 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:14.635806    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh" podStartSLOduration=1.635791244 podStartE2EDuration="1.635791244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:14.633437463 +0000 UTC m=+1543.001026906" watchObservedRunningTime="2026-09-11 14:18:14.635791244 +0000 UTC m=+1543.003380743"
Sep 11 14:18:22.728031 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:22.727991    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p" podUID="9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" containerName="main" containerID="cri-o://2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93" gracePeriod=45
Sep 11 14:18:22.983741 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:22.983664    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.021312    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tls-certs\") pod \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") "
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.021377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tmp-dir\") pod \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") "
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.021434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-home\") pod \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") "
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.021484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-dshm\") pod \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") "
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.021512    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-model-cache\") pod \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") "
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.021578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t28f\" (UniqueName: \"kubernetes.io/projected/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-kube-api-access-7t28f\") pod \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\" (UID: \"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41\") "
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.022459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" (UID: "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:23.022775 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.022363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-home" (OuterVolumeSpecName: "home") pod "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" (UID: "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:23.023423 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.022808    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" (UID: "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:23.024755 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.024724    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-dshm" (OuterVolumeSpecName: "dshm") pod "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" (UID: "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:23.025305 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.025284    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" (UID: "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:18:23.025768 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.025748    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-kube-api-access-7t28f" (OuterVolumeSpecName: "kube-api-access-7t28f") pod "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" (UID: "9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41"). InnerVolumeSpecName "kube-api-access-7t28f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:23.122652 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.122620    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t28f\" (UniqueName: \"kubernetes.io/projected/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-kube-api-access-7t28f\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:23.122652 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.122649    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:23.122652 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.122660    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:23.122894 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.122669    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:23.122894 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.122676    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:23.122894 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.122684    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:23.610503 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.610468    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:23.610687 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.610524    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:23.623112 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.623086    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:23.657946 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.657887    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" containerID="2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93" exitCode=0
Sep 11 14:18:23.658138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.657974    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"
Sep 11 14:18:23.658138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.657974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p" event={"ID":"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41","Type":"ContainerDied","Data":"2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93"}
Sep 11 14:18:23.658138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.658027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p" event={"ID":"9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41","Type":"ContainerDied","Data":"8420350e981534be18771bdc608c569c261b223ebbbaabe79024a89c8496c9e2"}
Sep 11 14:18:23.658138 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.658049    2561 scope.go:117] "RemoveContainer" containerID="2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93"
Sep 11 14:18:23.666852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.666834    2561 scope.go:117] "RemoveContainer" containerID="2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93"
Sep 11 14:18:23.667152 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:23.667133    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93\": container with ID starting with 2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93 not found: ID does not exist" containerID="2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93"
Sep 11 14:18:23.667238 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.667162    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93"} err="failed to get container status \"2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93\": rpc error: code = NotFound desc = could not find container \"2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93\": container with ID starting with 2efd0a0cad6f55cc338ed48ea129f16c40763fc0492bada26f3514e73fbefb93 not found: ID does not exist"
Sep 11 14:18:23.670360 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.670338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:18:23.683725 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.683692    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"]
Sep 11 14:18:23.689971 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.689949    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/sub-test-model-fb65d4-kserve-65cd895f89-4mb2p"]
Sep 11 14:18:23.716582 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.716541    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks" podUID="8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" containerName="main" containerID="cri-o://f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b" gracePeriod=45
Sep 11 14:18:23.960923 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:23.960887    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:18:24.031757 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.031723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tls-certs\") pod \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") "
Sep 11 14:18:24.031979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.031776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-home\") pod \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") "
Sep 11 14:18:24.031979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.031805    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tmp-dir\") pod \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") "
Sep 11 14:18:24.031979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.031832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-model-cache\") pod \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") "
Sep 11 14:18:24.031979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.031902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcpj2\" (UniqueName: \"kubernetes.io/projected/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-kube-api-access-bcpj2\") pod \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") "
Sep 11 14:18:24.031979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.031956    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-dshm\") pod \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\" (UID: \"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea\") "
Sep 11 14:18:24.032265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.032113    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-home" (OuterVolumeSpecName: "home") pod "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" (UID: "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:24.032265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.032128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" (UID: "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:24.032265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.032161    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" (UID: "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:24.032265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.032246    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:24.032265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.032261    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:24.032265 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.032271    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:24.034079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.034048    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-dshm" (OuterVolumeSpecName: "dshm") pod "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" (UID: "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:18:24.034188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.034125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" (UID: "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:18:24.034188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.034130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-kube-api-access-bcpj2" (OuterVolumeSpecName: "kube-api-access-bcpj2") pod "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" (UID: "8e2c484d-d7d7-49a4-85e7-b5936c3f9cea"). InnerVolumeSpecName "kube-api-access-bcpj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:18:24.118197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.118165    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" path="/var/lib/kubelet/pods/9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41/volumes"
Sep 11 14:18:24.133653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.133582    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:24.133653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.133609    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpj2\" (UniqueName: \"kubernetes.io/projected/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-kube-api-access-bcpj2\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:24.133653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.133624    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:18:24.666073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.666030    2561 generic.go:356] "Generic (PLEG): container finished" podID="8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" containerID="f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b" exitCode=0
Sep 11 14:18:24.666289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.666084    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"
Sep 11 14:18:24.666289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.666080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks" event={"ID":"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea","Type":"ContainerDied","Data":"f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b"}
Sep 11 14:18:24.666289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.666128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks" event={"ID":"8e2c484d-d7d7-49a4-85e7-b5936c3f9cea","Type":"ContainerDied","Data":"ef198d5d171da796035a6167464110228e9eb725ad509a5cdd19496e054d551d"}
Sep 11 14:18:24.666289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.666149    2561 scope.go:117] "RemoveContainer" containerID="f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b"
Sep 11 14:18:24.674505 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.674486    2561 scope.go:117] "RemoveContainer" containerID="f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b"
Sep 11 14:18:24.674790 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:18:24.674771    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b\": container with ID starting with f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b not found: ID does not exist" containerID="f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b"
Sep 11 14:18:24.674852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.674798    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b"} err="failed to get container status \"f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b\": rpc error: code = NotFound desc = could not find container \"f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b\": container with ID starting with f17822abb32323aa1f1c739b973ff4925430386478037be8e62126c7feaeb81b not found: ID does not exist"
Sep 11 14:18:24.690535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.690509    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"]
Sep 11 14:18:24.693325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:24.693306    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/sub-test-model-261f9c-kserve-784786bf4f-9tjks"]
Sep 11 14:18:26.116580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:26.116549    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" path="/var/lib/kubelet/pods/8e2c484d-d7d7-49a4-85e7-b5936c3f9cea/volumes"
Sep 11 14:18:28.611938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.611058    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"]
Sep 11 14:18:28.611938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.611748    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" containerName="main"
Sep 11 14:18:28.611938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.611770    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" containerName="main"
Sep 11 14:18:28.611938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.611798    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" containerName="main"
Sep 11 14:18:28.611938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.611807    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" containerName="main"
Sep 11 14:18:28.612690 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.612670    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b1eba2d-64c3-4dbb-9f0a-deaac28bfb41" containerName="main"
Sep 11 14:18:28.612796 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.612785    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2c484d-d7d7-49a4-85e7-b5936c3f9cea" containerName="main"
Sep 11 14:18:28.617968 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.617946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.621990 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.621938    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-106468\""
Sep 11 14:18:28.622550 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.622529    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-106468-dockercfg-2fdfn\""
Sep 11 14:18:28.624251 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.624225    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"]
Sep 11 14:18:28.673825 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.673786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae573fa4-a090-423f-bea5-f6b95b0bad26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.673825 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.673828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nh2k\" (UniqueName: \"kubernetes.io/projected/ae573fa4-a090-423f-bea5-f6b95b0bad26-kube-api-access-4nh2k\") pod \"payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.703051 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.703021    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"]
Sep 11 14:18:28.706357 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.706337    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.717104 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.717080    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"]
Sep 11 14:18:28.774709 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.774675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4k9s\" (UniqueName: \"kubernetes.io/projected/5dc1c078-b1a8-4e39-a5ad-45409714b231-kube-api-access-x4k9s\") pod \"payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") " pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.774875 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.774718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae573fa4-a090-423f-bea5-f6b95b0bad26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.774875 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.774746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nh2k\" (UniqueName: \"kubernetes.io/projected/ae573fa4-a090-423f-bea5-f6b95b0bad26-kube-api-access-4nh2k\") pod \"payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.774875 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.774827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc1c078-b1a8-4e39-a5ad-45409714b231-plugins-config-volume\") pod \"payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") " pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.775454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.775437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae573fa4-a090-423f-bea5-f6b95b0bad26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.788889 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.788863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nh2k\" (UniqueName: \"kubernetes.io/projected/ae573fa4-a090-423f-bea5-f6b95b0bad26-kube-api-access-4nh2k\") pod \"payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:28.875517 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.875435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k9s\" (UniqueName: \"kubernetes.io/projected/5dc1c078-b1a8-4e39-a5ad-45409714b231-kube-api-access-x4k9s\") pod \"payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") " pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.875673 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.875585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc1c078-b1a8-4e39-a5ad-45409714b231-plugins-config-volume\") pod \"payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") " pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.876244 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.876227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc1c078-b1a8-4e39-a5ad-45409714b231-plugins-config-volume\") pod \"payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") " pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.886546 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.886522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k9s\" (UniqueName: \"kubernetes.io/projected/5dc1c078-b1a8-4e39-a5ad-45409714b231-kube-api-access-x4k9s\") pod \"payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") " pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:28.928342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:28.928304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:29.016660 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.016631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:29.245724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.245635    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"]
Sep 11 14:18:29.249557 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:18:29.249533    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae573fa4_a090_423f_bea5_f6b95b0bad26.slice/crio-5142a5fb14b49b5c876bc8d0eaad11181e02f4deabc8f1b6b5a1f95bd6b1293e WatchSource:0}: Error finding container 5142a5fb14b49b5c876bc8d0eaad11181e02f4deabc8f1b6b5a1f95bd6b1293e: Status 404 returned error can't find the container with id 5142a5fb14b49b5c876bc8d0eaad11181e02f4deabc8f1b6b5a1f95bd6b1293e
Sep 11 14:18:29.315625 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.315584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"]
Sep 11 14:18:29.318079 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:18:29.318045    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc1c078_b1a8_4e39_a5ad_45409714b231.slice/crio-130e6ac95dc1df29555affe659859eeba28de1e5f36ec230a29316d5c8fef93d WatchSource:0}: Error finding container 130e6ac95dc1df29555affe659859eeba28de1e5f36ec230a29316d5c8fef93d: Status 404 returned error can't find the container with id 130e6ac95dc1df29555affe659859eeba28de1e5f36ec230a29316d5c8fef93d
Sep 11 14:18:29.691760 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.691721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9" event={"ID":"5dc1c078-b1a8-4e39-a5ad-45409714b231","Type":"ContainerStarted","Data":"31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84"}
Sep 11 14:18:29.692257 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.691779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9" event={"ID":"5dc1c078-b1a8-4e39-a5ad-45409714b231","Type":"ContainerStarted","Data":"130e6ac95dc1df29555affe659859eeba28de1e5f36ec230a29316d5c8fef93d"}
Sep 11 14:18:29.692257 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.691809    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:29.693422 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.693393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8" event={"ID":"ae573fa4-a090-423f-bea5-f6b95b0bad26","Type":"ContainerStarted","Data":"f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a"}
Sep 11 14:18:29.693529 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.693432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8" event={"ID":"ae573fa4-a090-423f-bea5-f6b95b0bad26","Type":"ContainerStarted","Data":"5142a5fb14b49b5c876bc8d0eaad11181e02f4deabc8f1b6b5a1f95bd6b1293e"}
Sep 11 14:18:29.693529 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.693453    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:29.728383 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.728332    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9" podStartSLOduration=1.728314728 podStartE2EDuration="1.728314728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:29.727798934 +0000 UTC m=+1558.095388380" watchObservedRunningTime="2026-09-11 14:18:29.728314728 +0000 UTC m=+1558.095904196"
Sep 11 14:18:29.759036 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:29.758980    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8" podStartSLOduration=1.758963485 podStartE2EDuration="1.758963485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:29.756125289 +0000 UTC m=+1558.123714735" watchObservedRunningTime="2026-09-11 14:18:29.758963485 +0000 UTC m=+1558.126553015"
Sep 11 14:18:32.340627 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:32.340598    2561 scope.go:117] "RemoveContainer" containerID="ce8afa3e6a522733d3614880e894727b72d597afa702021ec2d0f1ead2cb88b4"
Sep 11 14:18:33.347343 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:33.347299    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:18:40.700086 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.700045    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:18:40.700567 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.700338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:18:40.897453 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.897418    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"]
Sep 11 14:18:40.900746 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.900724    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:40.903715 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.903694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"default-dockercfg-qjcj7\""
Sep 11 14:18:40.904094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.904075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"rate-test-model-261f9c-kserve-self-signed-certs\""
Sep 11 14:18:40.904785 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.904758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"openshift-service-ca.crt\""
Sep 11 14:18:40.904893 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.904825    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-261f9c\"/\"kube-root-ca.crt\""
Sep 11 14:18:40.917168 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.917145    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"]
Sep 11 14:18:40.984395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.984303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csb9m\" (UniqueName: \"kubernetes.io/projected/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-kube-api-access-csb9m\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:40.984395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.984352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-home\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:40.984395 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.984372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-dshm\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:40.984633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.984399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tls-certs\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:40.984633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.984464    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-model-cache\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:40.984633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:40.984512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tmp-dir\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-model-cache\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tmp-dir\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csb9m\" (UniqueName: \"kubernetes.io/projected/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-kube-api-access-csb9m\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-home\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-dshm\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tls-certs\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085834 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-model-cache\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085834 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tmp-dir\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.085967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.085841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-home\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.087830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.087800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-dshm\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.088075 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.088058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tls-certs\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.094801 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.094778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csb9m\" (UniqueName: \"kubernetes.io/projected/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-kube-api-access-csb9m\") pod \"rate-test-model-261f9c-kserve-5648688b59-k4ct7\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") " pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.211279 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.211245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:41.489386 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.489356    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"]
Sep 11 14:18:41.491008 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:18:41.490966    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48fa6a0c_06a2_47fd_b5c9_9d35448fe87b.slice/crio-ac4f33e4feb50b0c4fe40b8e954da4f77700156a65375386437ab942a9ff8f21 WatchSource:0}: Error finding container ac4f33e4feb50b0c4fe40b8e954da4f77700156a65375386437ab942a9ff8f21: Status 404 returned error can't find the container with id ac4f33e4feb50b0c4fe40b8e954da4f77700156a65375386437ab942a9ff8f21
Sep 11 14:18:41.753724 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.753687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7" event={"ID":"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b","Type":"ContainerStarted","Data":"ee71651263bb892205df1c56a8a39ec529d9280c0f46bb7f7e18a1e2c89fcdfc"}
Sep 11 14:18:41.754102 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.753735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7" event={"ID":"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b","Type":"ContainerStarted","Data":"ac4f33e4feb50b0c4fe40b8e954da4f77700156a65375386437ab942a9ff8f21"}
Sep 11 14:18:41.777550 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:41.777457    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7" podStartSLOduration=1.777441855 podStartE2EDuration="1.777441855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:41.775263913 +0000 UTC m=+1570.142853356" watchObservedRunningTime="2026-09-11 14:18:41.777441855 +0000 UTC m=+1570.145031297"
Sep 11 14:18:50.335592 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:50.335550    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:18:51.212283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:51.212244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:51.212572 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:51.212545    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:51.229414 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:51.229386    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:18:51.811726 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:18:51.811693    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:19:10.085709 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.085677    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"]
Sep 11 14:19:10.086148 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.085875    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2" podUID="38be31a7-425b-4ed5-a4ba-e80331d8d64f" containerName="payload-pre-processing" containerID="cri-o://b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543" gracePeriod=30
Sep 11 14:19:10.335894 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.335827    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:19:10.347027 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.347002    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38be31a7-425b-4ed5-a4ba-e80331d8d64f-plugins-config-volume\") pod \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") "
Sep 11 14:19:10.347165 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.347055    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvbwz\" (UniqueName: \"kubernetes.io/projected/38be31a7-425b-4ed5-a4ba-e80331d8d64f-kube-api-access-cvbwz\") pod \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\" (UID: \"38be31a7-425b-4ed5-a4ba-e80331d8d64f\") "
Sep 11 14:19:10.347313 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.347292    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38be31a7-425b-4ed5-a4ba-e80331d8d64f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38be31a7-425b-4ed5-a4ba-e80331d8d64f" (UID: "38be31a7-425b-4ed5-a4ba-e80331d8d64f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:19:10.347382 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.347368    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38be31a7-425b-4ed5-a4ba-e80331d8d64f-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:10.349034 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.349008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38be31a7-425b-4ed5-a4ba-e80331d8d64f-kube-api-access-cvbwz" (OuterVolumeSpecName: "kube-api-access-cvbwz") pod "38be31a7-425b-4ed5-a4ba-e80331d8d64f" (UID: "38be31a7-425b-4ed5-a4ba-e80331d8d64f"). InnerVolumeSpecName "kube-api-access-cvbwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:10.448327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.448280    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvbwz\" (UniqueName: \"kubernetes.io/projected/38be31a7-425b-4ed5-a4ba-e80331d8d64f-kube-api-access-cvbwz\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:10.886658 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.886619    2561 generic.go:356] "Generic (PLEG): container finished" podID="38be31a7-425b-4ed5-a4ba-e80331d8d64f" containerID="b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543" exitCode=0
Sep 11 14:19:10.886836 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.886712    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"
Sep 11 14:19:10.886836 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.886713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2" event={"ID":"38be31a7-425b-4ed5-a4ba-e80331d8d64f","Type":"ContainerDied","Data":"b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543"}
Sep 11 14:19:10.886932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.886872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2" event={"ID":"38be31a7-425b-4ed5-a4ba-e80331d8d64f","Type":"ContainerDied","Data":"d5decd7d0566f52ac72acb4c24c0ef010c9f9d730fc4abba8f36648dee3be8b3"}
Sep 11 14:19:10.886932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.886897    2561 scope.go:117] "RemoveContainer" containerID="b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543"
Sep 11 14:19:10.895586 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.895567    2561 scope.go:117] "RemoveContainer" containerID="b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543"
Sep 11 14:19:10.895877 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:19:10.895858    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543\": container with ID starting with b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543 not found: ID does not exist" containerID="b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543"
Sep 11 14:19:10.895978 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.895884    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543"} err="failed to get container status \"b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543\": rpc error: code = NotFound desc = could not find container \"b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543\": container with ID starting with b9392ceabfe4f9a7343e590d3c09905aabf8aae8b1ddf77f3f93daf4ae20c543 not found: ID does not exist"
Sep 11 14:19:10.911055 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.911022    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"]
Sep 11 14:19:10.914629 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:10.914604    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d457-9756796f4-mxvm2"]
Sep 11 14:19:12.116797 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:12.116764    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38be31a7-425b-4ed5-a4ba-e80331d8d64f" path="/var/lib/kubelet/pods/38be31a7-425b-4ed5-a4ba-e80331d8d64f/volumes"
Sep 11 14:19:14.110492 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:14.110459    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:19:33.613470 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:33.613423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:19:34.264785 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.264750    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"]
Sep 11 14:19:34.265041 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.265003    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9" podUID="5dc1c078-b1a8-4e39-a5ad-45409714b231" containerName="payload-processing" containerID="cri-o://31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84" gracePeriod=30
Sep 11 14:19:34.320452 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.320417    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"]
Sep 11 14:19:34.320643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.320622    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8" podUID="ae573fa4-a090-423f-bea5-f6b95b0bad26" containerName="payload-pre-processing" containerID="cri-o://f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a" gracePeriod=30
Sep 11 14:19:34.594445 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.594413    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:19:34.596656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.596637    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:19:34.659411 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.659381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae573fa4-a090-423f-bea5-f6b95b0bad26-plugins-config-volume\") pod \"ae573fa4-a090-423f-bea5-f6b95b0bad26\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") "
Sep 11 14:19:34.659932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.659471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4k9s\" (UniqueName: \"kubernetes.io/projected/5dc1c078-b1a8-4e39-a5ad-45409714b231-kube-api-access-x4k9s\") pod \"5dc1c078-b1a8-4e39-a5ad-45409714b231\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") "
Sep 11 14:19:34.659932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.659497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc1c078-b1a8-4e39-a5ad-45409714b231-plugins-config-volume\") pod \"5dc1c078-b1a8-4e39-a5ad-45409714b231\" (UID: \"5dc1c078-b1a8-4e39-a5ad-45409714b231\") "
Sep 11 14:19:34.659932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.659517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nh2k\" (UniqueName: \"kubernetes.io/projected/ae573fa4-a090-423f-bea5-f6b95b0bad26-kube-api-access-4nh2k\") pod \"ae573fa4-a090-423f-bea5-f6b95b0bad26\" (UID: \"ae573fa4-a090-423f-bea5-f6b95b0bad26\") "
Sep 11 14:19:34.659932 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.659819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae573fa4-a090-423f-bea5-f6b95b0bad26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae573fa4-a090-423f-bea5-f6b95b0bad26" (UID: "ae573fa4-a090-423f-bea5-f6b95b0bad26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:19:34.660181 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.659932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc1c078-b1a8-4e39-a5ad-45409714b231-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5dc1c078-b1a8-4e39-a5ad-45409714b231" (UID: "5dc1c078-b1a8-4e39-a5ad-45409714b231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:19:34.661779 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.661751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae573fa4-a090-423f-bea5-f6b95b0bad26-kube-api-access-4nh2k" (OuterVolumeSpecName: "kube-api-access-4nh2k") pod "ae573fa4-a090-423f-bea5-f6b95b0bad26" (UID: "ae573fa4-a090-423f-bea5-f6b95b0bad26"). InnerVolumeSpecName "kube-api-access-4nh2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:34.662180 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.662161    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc1c078-b1a8-4e39-a5ad-45409714b231-kube-api-access-x4k9s" (OuterVolumeSpecName: "kube-api-access-x4k9s") pod "5dc1c078-b1a8-4e39-a5ad-45409714b231" (UID: "5dc1c078-b1a8-4e39-a5ad-45409714b231"). InnerVolumeSpecName "kube-api-access-x4k9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:34.760397 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.760344    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae573fa4-a090-423f-bea5-f6b95b0bad26-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:34.760397 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.760389    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4k9s\" (UniqueName: \"kubernetes.io/projected/5dc1c078-b1a8-4e39-a5ad-45409714b231-kube-api-access-x4k9s\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:34.760397 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.760402    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc1c078-b1a8-4e39-a5ad-45409714b231-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:34.760656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:34.760416    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nh2k\" (UniqueName: \"kubernetes.io/projected/ae573fa4-a090-423f-bea5-f6b95b0bad26-kube-api-access-4nh2k\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:35.004457 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.004370    2561 generic.go:356] "Generic (PLEG): container finished" podID="5dc1c078-b1a8-4e39-a5ad-45409714b231" containerID="31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84" exitCode=0
Sep 11 14:19:35.004457 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.004415    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"
Sep 11 14:19:35.004671 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.004462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9" event={"ID":"5dc1c078-b1a8-4e39-a5ad-45409714b231","Type":"ContainerDied","Data":"31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84"}
Sep 11 14:19:35.004671 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.004523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9" event={"ID":"5dc1c078-b1a8-4e39-a5ad-45409714b231","Type":"ContainerDied","Data":"130e6ac95dc1df29555affe659859eeba28de1e5f36ec230a29316d5c8fef93d"}
Sep 11 14:19:35.004671 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.004555    2561 scope.go:117] "RemoveContainer" containerID="31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84"
Sep 11 14:19:35.005899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.005877    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae573fa4-a090-423f-bea5-f6b95b0bad26" containerID="f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a" exitCode=0
Sep 11 14:19:35.006015 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.005943    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"
Sep 11 14:19:35.006015 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.005961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8" event={"ID":"ae573fa4-a090-423f-bea5-f6b95b0bad26","Type":"ContainerDied","Data":"f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a"}
Sep 11 14:19:35.006015 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.006006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8" event={"ID":"ae573fa4-a090-423f-bea5-f6b95b0bad26","Type":"ContainerDied","Data":"5142a5fb14b49b5c876bc8d0eaad11181e02f4deabc8f1b6b5a1f95bd6b1293e"}
Sep 11 14:19:35.014888 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.014853    2561 scope.go:117] "RemoveContainer" containerID="31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84"
Sep 11 14:19:35.016659 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:19:35.016620    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84\": container with ID starting with 31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84 not found: ID does not exist" containerID="31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84"
Sep 11 14:19:35.017403 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.017359    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84"} err="failed to get container status \"31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84\": rpc error: code = NotFound desc = could not find container \"31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84\": container with ID starting with 31b1a8275d2e2365379dfa0695ef884fcade5b496d0ad062f3bc251a8d845d84 not found: ID does not exist"
Sep 11 14:19:35.017523 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.017509    2561 scope.go:117] "RemoveContainer" containerID="f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a"
Sep 11 14:19:35.026149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.026129    2561 scope.go:117] "RemoveContainer" containerID="f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a"
Sep 11 14:19:35.026471 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:19:35.026451    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a\": container with ID starting with f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a not found: ID does not exist" containerID="f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a"
Sep 11 14:19:35.026521 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.026481    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a"} err="failed to get container status \"f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a\": rpc error: code = NotFound desc = could not find container \"f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a\": container with ID starting with f17c48a98d1ee7d78f67557cb355b69c66982713a6b774041d394a485b32227a not found: ID does not exist"
Sep 11 14:19:35.041206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.041168    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"]
Sep 11 14:19:35.054704 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.054671    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-106468-d9ff44cdc-pcfh9"]
Sep 11 14:19:35.073417 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.073382    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"]
Sep 11 14:19:35.077073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:35.077040    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-106468-6dfc858dcd-jllf8"]
Sep 11 14:19:36.117209 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:36.117170    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc1c078-b1a8-4e39-a5ad-45409714b231" path="/var/lib/kubelet/pods/5dc1c078-b1a8-4e39-a5ad-45409714b231/volumes"
Sep 11 14:19:36.117601 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:36.117530    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae573fa4-a090-423f-bea5-f6b95b0bad26" path="/var/lib/kubelet/pods/ae573fa4-a090-423f-bea5-f6b95b0bad26/volumes"
Sep 11 14:19:37.827470 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:37.827438    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"]
Sep 11 14:19:38.921149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:38.921109    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"]
Sep 11 14:19:44.640757 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:44.640716    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:19:49.558451 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558410    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"]
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558717    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38be31a7-425b-4ed5-a4ba-e80331d8d64f" containerName="payload-pre-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558729    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38be31a7-425b-4ed5-a4ba-e80331d8d64f" containerName="payload-pre-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558737    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc1c078-b1a8-4e39-a5ad-45409714b231" containerName="payload-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558743    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc1c078-b1a8-4e39-a5ad-45409714b231" containerName="payload-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558749    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae573fa4-a090-423f-bea5-f6b95b0bad26" containerName="payload-pre-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558755    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae573fa4-a090-423f-bea5-f6b95b0bad26" containerName="payload-pre-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558818    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38be31a7-425b-4ed5-a4ba-e80331d8d64f" containerName="payload-pre-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558826    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc1c078-b1a8-4e39-a5ad-45409714b231" containerName="payload-processing"
Sep 11 14:19:49.558931 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.558835    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae573fa4-a090-423f-bea5-f6b95b0bad26" containerName="payload-pre-processing"
Sep 11 14:19:49.563100 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.563084    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.567464 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.567443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e6521d-dockercfg-sxwlg\""
Sep 11 14:19:49.567759 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.567745    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e6521d\""
Sep 11 14:19:49.580513 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.580489    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"]
Sep 11 14:19:49.596110 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.596070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhfcn\" (UniqueName: \"kubernetes.io/projected/20c798c7-5445-4cb2-88e4-2b25478bd892-kube-api-access-vhfcn\") pod \"payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.596245 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.596171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c798c7-5445-4cb2-88e4-2b25478bd892-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.697008 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.696965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhfcn\" (UniqueName: \"kubernetes.io/projected/20c798c7-5445-4cb2-88e4-2b25478bd892-kube-api-access-vhfcn\") pod \"payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.697219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.697071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c798c7-5445-4cb2-88e4-2b25478bd892-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.697816 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.697794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c798c7-5445-4cb2-88e4-2b25478bd892-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.719289 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.719262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhfcn\" (UniqueName: \"kubernetes.io/projected/20c798c7-5445-4cb2-88e4-2b25478bd892-kube-api-access-vhfcn\") pod \"payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:49.873967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:49.873846    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:50.153281 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:19:50.153246    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c798c7_5445_4cb2_88e4_2b25478bd892.slice/crio-cffe09598dfc5c6decfa15ef25753ba31490beef392b5f2dde568b8beef621da WatchSource:0}: Error finding container cffe09598dfc5c6decfa15ef25753ba31490beef392b5f2dde568b8beef621da: Status 404 returned error can't find the container with id cffe09598dfc5c6decfa15ef25753ba31490beef392b5f2dde568b8beef621da
Sep 11 14:19:50.155090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:50.155062    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"]
Sep 11 14:19:51.085980 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:51.085903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" event={"ID":"20c798c7-5445-4cb2-88e4-2b25478bd892","Type":"ContainerStarted","Data":"cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b"}
Sep 11 14:19:51.086412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:51.086011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:19:51.086412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:51.086037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" event={"ID":"20c798c7-5445-4cb2-88e4-2b25478bd892","Type":"ContainerStarted","Data":"cffe09598dfc5c6decfa15ef25753ba31490beef392b5f2dde568b8beef621da"}
Sep 11 14:19:51.141549 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:51.141493    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" podStartSLOduration=2.141475313 podStartE2EDuration="2.141475313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:51.139700111 +0000 UTC m=+1639.507289554" watchObservedRunningTime="2026-09-11 14:19:51.141475313 +0000 UTC m=+1639.509064755"
Sep 11 14:19:52.845954 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:52.845887    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh" podUID="1b1efd8b-e438-4e28-b48f-73260c652462" containerName="main" containerID="cri-o://c8a150123fa6e2701554a8a3c828805ae4f66cc38477adddb4ce75a43889799c" gracePeriod=45
Sep 11 14:19:53.097154 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.097079    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b1efd8b-e438-4e28-b48f-73260c652462" containerID="c8a150123fa6e2701554a8a3c828805ae4f66cc38477adddb4ce75a43889799c" exitCode=0
Sep 11 14:19:53.097306 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.097145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh" event={"ID":"1b1efd8b-e438-4e28-b48f-73260c652462","Type":"ContainerDied","Data":"c8a150123fa6e2701554a8a3c828805ae4f66cc38477adddb4ce75a43889799c"}
Sep 11 14:19:53.097306 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.097183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh" event={"ID":"1b1efd8b-e438-4e28-b48f-73260c652462","Type":"ContainerDied","Data":"b27883b517004d8030b0999fe60445cb34cb175ec14e9920a5ebc53b6ec3369e"}
Sep 11 14:19:53.097306 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.097203    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b27883b517004d8030b0999fe60445cb34cb175ec14e9920a5ebc53b6ec3369e"
Sep 11 14:19:53.106578 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.106554    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:19:53.122633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.122605    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-model-cache\") pod \"1b1efd8b-e438-4e28-b48f-73260c652462\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") "
Sep 11 14:19:53.122858 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.122837    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b1efd8b-e438-4e28-b48f-73260c652462" (UID: "1b1efd8b-e438-4e28-b48f-73260c652462"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:53.223373 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-tmp-dir\") pod \"1b1efd8b-e438-4e28-b48f-73260c652462\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") "
Sep 11 14:19:53.223373 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223374    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1efd8b-e438-4e28-b48f-73260c652462-tls-certs\") pod \"1b1efd8b-e438-4e28-b48f-73260c652462\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") "
Sep 11 14:19:53.223608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-dshm\") pod \"1b1efd8b-e438-4e28-b48f-73260c652462\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") "
Sep 11 14:19:53.223608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgqnr\" (UniqueName: \"kubernetes.io/projected/1b1efd8b-e438-4e28-b48f-73260c652462-kube-api-access-xgqnr\") pod \"1b1efd8b-e438-4e28-b48f-73260c652462\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") "
Sep 11 14:19:53.223608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223557    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-home\") pod \"1b1efd8b-e438-4e28-b48f-73260c652462\" (UID: \"1b1efd8b-e438-4e28-b48f-73260c652462\") "
Sep 11 14:19:53.223778 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b1efd8b-e438-4e28-b48f-73260c652462" (UID: "1b1efd8b-e438-4e28-b48f-73260c652462"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:53.223838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223801    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:53.223838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223821    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:53.223977 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.223840    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-home" (OuterVolumeSpecName: "home") pod "1b1efd8b-e438-4e28-b48f-73260c652462" (UID: "1b1efd8b-e438-4e28-b48f-73260c652462"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:53.225567 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.225546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-dshm" (OuterVolumeSpecName: "dshm") pod "1b1efd8b-e438-4e28-b48f-73260c652462" (UID: "1b1efd8b-e438-4e28-b48f-73260c652462"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:53.225567 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.225563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1efd8b-e438-4e28-b48f-73260c652462-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b1efd8b-e438-4e28-b48f-73260c652462" (UID: "1b1efd8b-e438-4e28-b48f-73260c652462"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:53.225784 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.225609    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1efd8b-e438-4e28-b48f-73260c652462-kube-api-access-xgqnr" (OuterVolumeSpecName: "kube-api-access-xgqnr") pod "1b1efd8b-e438-4e28-b48f-73260c652462" (UID: "1b1efd8b-e438-4e28-b48f-73260c652462"). InnerVolumeSpecName "kube-api-access-xgqnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:53.324207 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.324167    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1efd8b-e438-4e28-b48f-73260c652462-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:53.324207 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.324200    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:53.324207 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.324210    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgqnr\" (UniqueName: \"kubernetes.io/projected/1b1efd8b-e438-4e28-b48f-73260c652462-kube-api-access-xgqnr\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:53.324460 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.324220    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1efd8b-e438-4e28-b48f-73260c652462-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:53.939505 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:53.939465    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7" podUID="48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" containerName="main" containerID="cri-o://ee71651263bb892205df1c56a8a39ec529d9280c0f46bb7f7e18a1e2c89fcdfc" gracePeriod=45
Sep 11 14:19:54.103493 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.103460    2561 generic.go:356] "Generic (PLEG): container finished" podID="48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" containerID="ee71651263bb892205df1c56a8a39ec529d9280c0f46bb7f7e18a1e2c89fcdfc" exitCode=0
Sep 11 14:19:54.103681 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.103534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7" event={"ID":"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b","Type":"ContainerDied","Data":"ee71651263bb892205df1c56a8a39ec529d9280c0f46bb7f7e18a1e2c89fcdfc"}
Sep 11 14:19:54.103681 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.103640    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"
Sep 11 14:19:54.138072 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.138038    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"]
Sep 11 14:19:54.151478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.151444    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fb65d4/rate-test-model-fb65d4-kserve-54d4c7b59f-kcclh"]
Sep 11 14:19:54.190759 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.190699    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:19:54.230537 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-home\") pod \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") "
Sep 11 14:19:54.230637 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230546    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tmp-dir\") pod \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") "
Sep 11 14:19:54.230637 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230570    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tls-certs\") pod \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") "
Sep 11 14:19:54.230637 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csb9m\" (UniqueName: \"kubernetes.io/projected/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-kube-api-access-csb9m\") pod \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") "
Sep 11 14:19:54.230637 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230624    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-dshm\") pod \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") "
Sep 11 14:19:54.230824 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-model-cache\") pod \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\" (UID: \"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b\") "
Sep 11 14:19:54.230878 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230824    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" (UID: "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:54.230878 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.230857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-home" (OuterVolumeSpecName: "home") pod "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" (UID: "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:54.231046 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.231024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-model-cache" (OuterVolumeSpecName: "model-cache") pod "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" (UID: "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:54.232788 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.232769    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-dshm" (OuterVolumeSpecName: "dshm") pod "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" (UID: "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:19:54.232872 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.232850    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" (UID: "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:19:54.232935 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.232900    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-kube-api-access-csb9m" (OuterVolumeSpecName: "kube-api-access-csb9m") pod "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" (UID: "48fa6a0c-06a2-47fd-b5c9-9d35448fe87b"). InnerVolumeSpecName "kube-api-access-csb9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:54.331577 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.331539    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csb9m\" (UniqueName: \"kubernetes.io/projected/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-kube-api-access-csb9m\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:54.331577 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.331569    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:54.331577 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.331581    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:54.331577 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.331589    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:54.331852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.331599    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:54.331852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.331607    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:54.655264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.655223    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"]
Sep 11 14:19:54.655494 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.655470    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4" podUID="9115c6ef-7ed7-4c8b-b62e-e5da49684d47" containerName="payload-processing" containerID="cri-o://3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c" gracePeriod=30
Sep 11 14:19:54.892083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.892055    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:19:54.935538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.935460    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-plugins-config-volume\") pod \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") "
Sep 11 14:19:54.935538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.935507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2txtj\" (UniqueName: \"kubernetes.io/projected/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-kube-api-access-2txtj\") pod \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\" (UID: \"9115c6ef-7ed7-4c8b-b62e-e5da49684d47\") "
Sep 11 14:19:54.935859 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.935838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9115c6ef-7ed7-4c8b-b62e-e5da49684d47" (UID: "9115c6ef-7ed7-4c8b-b62e-e5da49684d47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:19:54.937902 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:54.937749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-kube-api-access-2txtj" (OuterVolumeSpecName: "kube-api-access-2txtj") pod "9115c6ef-7ed7-4c8b-b62e-e5da49684d47" (UID: "9115c6ef-7ed7-4c8b-b62e-e5da49684d47"). InnerVolumeSpecName "kube-api-access-2txtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:19:55.036201 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.036159    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:55.036201 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.036196    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txtj\" (UniqueName: \"kubernetes.io/projected/9115c6ef-7ed7-4c8b-b62e-e5da49684d47-kube-api-access-2txtj\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:19:55.109332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.109300    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"
Sep 11 14:19:55.109332 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.109325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7" event={"ID":"48fa6a0c-06a2-47fd-b5c9-9d35448fe87b","Type":"ContainerDied","Data":"ac4f33e4feb50b0c4fe40b8e954da4f77700156a65375386437ab942a9ff8f21"}
Sep 11 14:19:55.109608 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.109381    2561 scope.go:117] "RemoveContainer" containerID="ee71651263bb892205df1c56a8a39ec529d9280c0f46bb7f7e18a1e2c89fcdfc"
Sep 11 14:19:55.110818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.110707    2561 generic.go:356] "Generic (PLEG): container finished" podID="9115c6ef-7ed7-4c8b-b62e-e5da49684d47" containerID="3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c" exitCode=0
Sep 11 14:19:55.110818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.110753    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"
Sep 11 14:19:55.110975 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.110753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4" event={"ID":"9115c6ef-7ed7-4c8b-b62e-e5da49684d47","Type":"ContainerDied","Data":"3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c"}
Sep 11 14:19:55.110975 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.110903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4" event={"ID":"9115c6ef-7ed7-4c8b-b62e-e5da49684d47","Type":"ContainerDied","Data":"95bf7d5f78fc512876bdfbc51f1267515d4945a719d1b429315c1e45908cc826"}
Sep 11 14:19:55.119161 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.119026    2561 scope.go:117] "RemoveContainer" containerID="3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c"
Sep 11 14:19:55.126845 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.126829    2561 scope.go:117] "RemoveContainer" containerID="3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c"
Sep 11 14:19:55.127137 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:19:55.127116    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c\": container with ID starting with 3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c not found: ID does not exist" containerID="3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c"
Sep 11 14:19:55.127231 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.127145    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c"} err="failed to get container status \"3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c\": rpc error: code = NotFound desc = could not find container \"3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c\": container with ID starting with 3d0195e70a880fcb762cec0c4654630511d075d01b9bbd963ca8787a183bde5c not found: ID does not exist"
Sep 11 14:19:55.142721 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.142683    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"]
Sep 11 14:19:55.148299 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.148271    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c52865-594ffd9475-7pfh4"]
Sep 11 14:19:55.166032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.166003    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"]
Sep 11 14:19:55.183127 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:55.183094    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-261f9c/rate-test-model-261f9c-kserve-5648688b59-k4ct7"]
Sep 11 14:19:56.116926 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:56.116881    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1efd8b-e438-4e28-b48f-73260c652462" path="/var/lib/kubelet/pods/1b1efd8b-e438-4e28-b48f-73260c652462/volumes"
Sep 11 14:19:56.117314 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:56.117304    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" path="/var/lib/kubelet/pods/48fa6a0c-06a2-47fd-b5c9-9d35448fe87b/volumes"
Sep 11 14:19:56.117616 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:19:56.117603    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9115c6ef-7ed7-4c8b-b62e-e5da49684d47" path="/var/lib/kubelet/pods/9115c6ef-7ed7-4c8b-b62e-e5da49684d47/volumes"
Sep 11 14:20:01.323341 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:01.323307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:20:02.093679 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:02.093649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:20:03.469062 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469023    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"]
Sep 11 14:20:03.469454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469371    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9115c6ef-7ed7-4c8b-b62e-e5da49684d47" containerName="payload-processing"
Sep 11 14:20:03.469454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469383    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9115c6ef-7ed7-4c8b-b62e-e5da49684d47" containerName="payload-processing"
Sep 11 14:20:03.469454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469391    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1efd8b-e438-4e28-b48f-73260c652462" containerName="main"
Sep 11 14:20:03.469454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469396    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1efd8b-e438-4e28-b48f-73260c652462" containerName="main"
Sep 11 14:20:03.469454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469411    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" containerName="main"
Sep 11 14:20:03.469454 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469417    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" containerName="main"
Sep 11 14:20:03.469656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469464    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9115c6ef-7ed7-4c8b-b62e-e5da49684d47" containerName="payload-processing"
Sep 11 14:20:03.469656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469473    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1efd8b-e438-4e28-b48f-73260c652462" containerName="main"
Sep 11 14:20:03.469656 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.469480    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fa6a0c-06a2-47fd-b5c9-9d35448fe87b" containerName="main"
Sep 11 14:20:03.472296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.472281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.475590 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.475568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d9cd47-openshift-default-dockercfg-wrd4f\""
Sep 11 14:20:03.491206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.491178    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"]
Sep 11 14:20:03.492866 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.492834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-podinfo\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-certs\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-credential-socket\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-data\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbb8k\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-kube-api-access-kbb8k\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-socket\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493327 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd10f49-bd76-441e-abf6-5b0d8c112275-istiod-ca-cert\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493676 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-envoy\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.493676 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.493380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-token\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594322 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-socket\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594322 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd10f49-bd76-441e-abf6-5b0d8c112275-istiod-ca-cert\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-envoy\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-token\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-podinfo\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-certs\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-credential-socket\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-data\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbb8k\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-kube-api-access-kbb8k\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-socket\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-certs\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.594950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-credential-socket\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.595075 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.594988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-data\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.595198 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.595169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd10f49-bd76-441e-abf6-5b0d8c112275-istiod-ca-cert\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.596752 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.596732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-envoy\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.596987 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.596967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-podinfo\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.606235 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.606212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbb8k\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-kube-api-access-kbb8k\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.615205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.615177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-token\") pod \"e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") " pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.782625 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.782536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:03.883224 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.883190    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"]
Sep 11 14:20:03.883566 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:03.883416    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy" containerID="cri-o://d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b" gracePeriod=30
Sep 11 14:20:04.114699 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:20:04.114669    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd10f49_bd76_441e_abf6_5b0d8c112275.slice/crio-dd0fda41c3bdc34f95de99b5a47f5b2d5d16d85eae245fbf2557b6e825939c56 WatchSource:0}: Error finding container dd0fda41c3bdc34f95de99b5a47f5b2d5d16d85eae245fbf2557b6e825939c56: Status 404 returned error can't find the container with id dd0fda41c3bdc34f95de99b5a47f5b2d5d16d85eae245fbf2557b6e825939c56
Sep 11 14:20:04.116871 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.116833    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:20:04.116967 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.116944    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:20:04.117009 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.116986    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:20:04.117717 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.117686    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"]
Sep 11 14:20:04.153376 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.153348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" event={"ID":"8bd10f49-bd76-441e-abf6-5b0d8c112275","Type":"ContainerStarted","Data":"dd0fda41c3bdc34f95de99b5a47f5b2d5d16d85eae245fbf2557b6e825939c56"}
Sep 11 14:20:04.296328 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.296295    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"]
Sep 11 14:20:04.299646 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.299626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:04.302612 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.302589    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:20:04.302975 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.302958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:20:04.314512 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.314468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"]
Sep 11 14:20:04.401202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.401108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e95f6f-5c9e-42ba-8596-76a036f4638c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9\" (UID: \"41e95f6f-5c9e-42ba-8596-76a036f4638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:04.502587 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.502537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e95f6f-5c9e-42ba-8596-76a036f4638c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9\" (UID: \"41e95f6f-5c9e-42ba-8596-76a036f4638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:04.503170 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.503151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e95f6f-5c9e-42ba-8596-76a036f4638c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9\" (UID: \"41e95f6f-5c9e-42ba-8596-76a036f4638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:04.611283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.611252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:04.887425 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:04.887394    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"]
Sep 11 14:20:04.890814 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:20:04.890782    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e95f6f_5c9e_42ba_8596_76a036f4638c.slice/crio-55adf4c92693478f6ec46008e7fbdfbc2ffd2a80f0ebb7ab253813c22e1b8663 WatchSource:0}: Error finding container 55adf4c92693478f6ec46008e7fbdfbc2ffd2a80f0ebb7ab253813c22e1b8663: Status 404 returned error can't find the container with id 55adf4c92693478f6ec46008e7fbdfbc2ffd2a80f0ebb7ab253813c22e1b8663
Sep 11 14:20:05.160367 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:05.160334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" event={"ID":"8bd10f49-bd76-441e-abf6-5b0d8c112275","Type":"ContainerStarted","Data":"926f9818ff532ed0acc2d6605f4eee56393d698a871e84c46a31dd9087280586"}
Sep 11 14:20:05.161778 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:05.161754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9" event={"ID":"41e95f6f-5c9e-42ba-8596-76a036f4638c","Type":"ContainerStarted","Data":"55adf4c92693478f6ec46008e7fbdfbc2ffd2a80f0ebb7ab253813c22e1b8663"}
Sep 11 14:20:05.190576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:05.190485    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podStartSLOduration=2.190469416 podStartE2EDuration="2.190469416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:05.18933126 +0000 UTC m=+1653.556920704" watchObservedRunningTime="2026-09-11 14:20:05.190469416 +0000 UTC m=+1653.558058859"
Sep 11 14:20:05.783339 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:05.783304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:05.788403 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:05.788377    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:06.167218 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:06.167179    2561 generic.go:356] "Generic (PLEG): container finished" podID="41e95f6f-5c9e-42ba-8596-76a036f4638c" containerID="89a30de86a33ee1e60ea1ccf44025cabd84822d8584b877cd9577950bfb42492" exitCode=0
Sep 11 14:20:06.167402 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:06.167263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9" event={"ID":"41e95f6f-5c9e-42ba-8596-76a036f4638c","Type":"ContainerDied","Data":"89a30de86a33ee1e60ea1ccf44025cabd84822d8584b877cd9577950bfb42492"}
Sep 11 14:20:06.167618 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:06.167596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:06.168480 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:06.168463    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:07.305720 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:07.305686    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:07.330091 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:07.330057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e95f6f-5c9e-42ba-8596-76a036f4638c-maas-api-service-ca\") pod \"41e95f6f-5c9e-42ba-8596-76a036f4638c\" (UID: \"41e95f6f-5c9e-42ba-8596-76a036f4638c\") "
Sep 11 14:20:07.330471 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:07.330445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e95f6f-5c9e-42ba-8596-76a036f4638c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41e95f6f-5c9e-42ba-8596-76a036f4638c" (UID: "41e95f6f-5c9e-42ba-8596-76a036f4638c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:07.431653 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:07.431572    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e95f6f-5c9e-42ba-8596-76a036f4638c-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:08.178148 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:08.178114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9" event={"ID":"41e95f6f-5c9e-42ba-8596-76a036f4638c","Type":"ContainerDied","Data":"55adf4c92693478f6ec46008e7fbdfbc2ffd2a80f0ebb7ab253813c22e1b8663"}
Sep 11 14:20:08.178148 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:08.178132    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"
Sep 11 14:20:08.178401 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:08.178162    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55adf4c92693478f6ec46008e7fbdfbc2ffd2a80f0ebb7ab253813c22e1b8663"
Sep 11 14:20:09.136083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.136055    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:20:09.184303 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.184223    2561 generic.go:356] "Generic (PLEG): container finished" podID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerID="d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b" exitCode=0
Sep 11 14:20:09.184303 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.184273    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"
Sep 11 14:20:09.184303 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.184283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" event={"ID":"7f94f2f2-c2de-4750-8998-551fd21bf9d4","Type":"ContainerDied","Data":"d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b"}
Sep 11 14:20:09.184524 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.184330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd" event={"ID":"7f94f2f2-c2de-4750-8998-551fd21bf9d4","Type":"ContainerDied","Data":"01760fcc7f73f0f477fcb4844c7a82fb2cc1457c0caf2a40b392f7c0f4f3e913"}
Sep 11 14:20:09.184524 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.184356    2561 scope.go:117] "RemoveContainer" containerID="d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b"
Sep 11 14:20:09.193149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.193129    2561 scope.go:117] "RemoveContainer" containerID="42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"
Sep 11 14:20:09.200571 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.200553    2561 scope.go:117] "RemoveContainer" containerID="d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b"
Sep 11 14:20:09.200865 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:20:09.200843    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b\": container with ID starting with d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b not found: ID does not exist" containerID="d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b"
Sep 11 14:20:09.200978 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.200871    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b"} err="failed to get container status \"d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b\": rpc error: code = NotFound desc = could not find container \"d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b\": container with ID starting with d0795e367ef19bb4a48c4f33fef33c5859f577ffe2b78a28dd12940ff66fc21b not found: ID does not exist"
Sep 11 14:20:09.200978 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.200893    2561 scope.go:117] "RemoveContainer" containerID="42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"
Sep 11 14:20:09.201167 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:20:09.201154    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41\": container with ID starting with 42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41 not found: ID does not exist" containerID="42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"
Sep 11 14:20:09.201223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.201171    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41"} err="failed to get container status \"42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41\": rpc error: code = NotFound desc = could not find container \"42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41\": container with ID starting with 42ada29b227dafb45f644da73db89a9f66e7bb2b692aeca2457594b03a85cb41 not found: ID does not exist"
Sep 11 14:20:09.247349 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istiod-ca-cert\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-token\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247398    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-credential-socket\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247454    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-socket\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-podinfo\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-certs\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247560    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2rbx\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-kube-api-access-l2rbx\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247584    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-data\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-envoy\") pod \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\" (UID: \"7f94f2f2-c2de-4750-8998-551fd21bf9d4\") "
Sep 11 14:20:09.247968 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247821    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:20:09.247968 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:20:09.247968 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.247848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:09.248145 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.248067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:20:09.248145 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.248094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:20:09.249677 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.249649    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:20:09.249818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.249803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:20:09.250087 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.250063    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:09.250162 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.250080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-kube-api-access-l2rbx" (OuterVolumeSpecName: "kube-api-access-l2rbx") pod "7f94f2f2-c2de-4750-8998-551fd21bf9d4" (UID: "7f94f2f2-c2de-4750-8998-551fd21bf9d4"). InnerVolumeSpecName "kube-api-access-l2rbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:09.348534 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348502    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-podinfo\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348534 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348532    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348545    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2rbx\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-kube-api-access-l2rbx\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348558    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-data\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348571    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-envoy\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348581    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istiod-ca-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348592    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f94f2f2-c2de-4750-8998-551fd21bf9d4-istio-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348603    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-credential-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.348749 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.348615    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f94f2f2-c2de-4750-8998-551fd21bf9d4-workload-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.521900 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.521870    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"]
Sep 11 14:20:09.537158 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:09.537131    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c52865-openshift-default-bf8cf6976-pcvjd"]
Sep 11 14:20:10.116816 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:10.116783    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" path="/var/lib/kubelet/pods/7f94f2f2-c2de-4750-8998-551fd21bf9d4/volumes"
Sep 11 14:20:11.852210 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852173    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"]
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852497    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy"
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852507    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy"
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e95f6f-5c9e-42ba-8596-76a036f4638c" containerName="revoke-keys"
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852539    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e95f6f-5c9e-42ba-8596-76a036f4638c" containerName="revoke-keys"
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852586    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy"
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852593    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e95f6f-5c9e-42ba-8596-76a036f4638c" containerName="revoke-keys"
Sep 11 14:20:11.852605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.852600    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy"
Sep 11 14:20:11.856728 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.856704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:11.860122 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.860099    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d9cd47\""
Sep 11 14:20:11.884442 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.884412    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"]
Sep 11 14:20:11.924140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.924108    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"]
Sep 11 14:20:11.924442 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.924430    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy"
Sep 11 14:20:11.924502 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.924443    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f94f2f2-c2de-4750-8998-551fd21bf9d4" containerName="istio-proxy"
Sep 11 14:20:11.927422 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.927404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:11.932084 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.932066    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d9cd47-dockercfg-j8blp\""
Sep 11 14:20:11.942997 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.942973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"]
Sep 11 14:20:11.981038 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.981000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:11.981197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.981138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1b244-87d4-4164-9a1a-2dc3c8941408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:11.981262 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.981201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t725b\" (UniqueName: \"kubernetes.io/projected/f0b1b244-87d4-4164-9a1a-2dc3c8941408-kube-api-access-t725b\") pod \"payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:11.981262 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:11.981232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6k2\" (UniqueName: \"kubernetes.io/projected/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-kube-api-access-9x6k2\") pod \"payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:12.082578 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.082535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1b244-87d4-4164-9a1a-2dc3c8941408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:12.082578 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.082585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t725b\" (UniqueName: \"kubernetes.io/projected/f0b1b244-87d4-4164-9a1a-2dc3c8941408-kube-api-access-t725b\") pod \"payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:12.082807 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.082631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6k2\" (UniqueName: \"kubernetes.io/projected/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-kube-api-access-9x6k2\") pod \"payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:12.082807 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.082654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:12.083605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.083581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:12.083667 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.083581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1b244-87d4-4164-9a1a-2dc3c8941408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:12.092575 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.092548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t725b\" (UniqueName: \"kubernetes.io/projected/f0b1b244-87d4-4164-9a1a-2dc3c8941408-kube-api-access-t725b\") pod \"payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:12.092695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.092661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6k2\" (UniqueName: \"kubernetes.io/projected/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-kube-api-access-9x6k2\") pod \"payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:12.166730 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.166640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:12.237274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.237245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:12.456461 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.456383    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"]
Sep 11 14:20:12.460854 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:20:12.460823    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0dfb4f_5789_44a6_8d5a_4cc0848a0376.slice/crio-25ab8278bdf0ac7ccb4b65958072f43816b53d307b738740d38bdb2fa9995586 WatchSource:0}: Error finding container 25ab8278bdf0ac7ccb4b65958072f43816b53d307b738740d38bdb2fa9995586: Status 404 returned error can't find the container with id 25ab8278bdf0ac7ccb4b65958072f43816b53d307b738740d38bdb2fa9995586
Sep 11 14:20:12.520580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:12.520540    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"]
Sep 11 14:20:12.523441 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:20:12.523414    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b1b244_87d4_4164_9a1a_2dc3c8941408.slice/crio-2ef08dbbe272f671a544015718a9bae4851d51c5a6eed12c3c7b085a963e6972 WatchSource:0}: Error finding container 2ef08dbbe272f671a544015718a9bae4851d51c5a6eed12c3c7b085a963e6972: Status 404 returned error can't find the container with id 2ef08dbbe272f671a544015718a9bae4851d51c5a6eed12c3c7b085a963e6972
Sep 11 14:20:13.206188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.206145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j" event={"ID":"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376","Type":"ContainerStarted","Data":"54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95"}
Sep 11 14:20:13.206691 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.206215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j" event={"ID":"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376","Type":"ContainerStarted","Data":"25ab8278bdf0ac7ccb4b65958072f43816b53d307b738740d38bdb2fa9995586"}
Sep 11 14:20:13.206691 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.206248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:13.207818 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.207795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc" event={"ID":"f0b1b244-87d4-4164-9a1a-2dc3c8941408","Type":"ContainerStarted","Data":"f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda"}
Sep 11 14:20:13.207945 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.207848    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:13.207945 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.207875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc" event={"ID":"f0b1b244-87d4-4164-9a1a-2dc3c8941408","Type":"ContainerStarted","Data":"2ef08dbbe272f671a544015718a9bae4851d51c5a6eed12c3c7b085a963e6972"}
Sep 11 14:20:13.250839 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:13.250777    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j" podStartSLOduration=2.250755851 podStartE2EDuration="2.250755851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:13.249261325 +0000 UTC m=+1661.616850767" watchObservedRunningTime="2026-09-11 14:20:13.250755851 +0000 UTC m=+1661.618345298"
Sep 11 14:20:14.937347 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:14.937287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc" podStartSLOduration=3.937267972 podStartE2EDuration="3.937267972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:13.297553086 +0000 UTC m=+1661.665142530" watchObservedRunningTime="2026-09-11 14:20:14.937267972 +0000 UTC m=+1663.304857417"
Sep 11 14:20:14.938166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:14.938146    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:20:16.393817 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.393774    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"]
Sep 11 14:20:16.404467 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.404436    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5cf565-29818935-dzmcq"]
Sep 11 14:20:16.470833 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.470796    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"]
Sep 11 14:20:16.471128 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.471100    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n" podUID="fa5cf026-ae2e-4204-a1be-b67217d18de8" containerName="payload-pre-processing" containerID="cri-o://588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22" gracePeriod=30
Sep 11 14:20:16.696250 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.696222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"]
Sep 11 14:20:16.699140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.699111    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5cf565-l6js9"]
Sep 11 14:20:16.770225 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.770195    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:20:16.840255 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.840186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5cf026-ae2e-4204-a1be-b67217d18de8-plugins-config-volume\") pod \"fa5cf026-ae2e-4204-a1be-b67217d18de8\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") "
Sep 11 14:20:16.841218 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.840553    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9smg\" (UniqueName: \"kubernetes.io/projected/fa5cf026-ae2e-4204-a1be-b67217d18de8-kube-api-access-x9smg\") pod \"fa5cf026-ae2e-4204-a1be-b67217d18de8\" (UID: \"fa5cf026-ae2e-4204-a1be-b67217d18de8\") "
Sep 11 14:20:16.844166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.844136    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5cf026-ae2e-4204-a1be-b67217d18de8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa5cf026-ae2e-4204-a1be-b67217d18de8" (UID: "fa5cf026-ae2e-4204-a1be-b67217d18de8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:16.847080 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.847050    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5cf026-ae2e-4204-a1be-b67217d18de8-kube-api-access-x9smg" (OuterVolumeSpecName: "kube-api-access-x9smg") pod "fa5cf026-ae2e-4204-a1be-b67217d18de8" (UID: "fa5cf026-ae2e-4204-a1be-b67217d18de8"). InnerVolumeSpecName "kube-api-access-x9smg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:16.942450 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.942339    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa5cf026-ae2e-4204-a1be-b67217d18de8-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:16.942450 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:16.942385    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9smg\" (UniqueName: \"kubernetes.io/projected/fa5cf026-ae2e-4204-a1be-b67217d18de8-kube-api-access-x9smg\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:20:17.231985 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.231869    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa5cf026-ae2e-4204-a1be-b67217d18de8" containerID="588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22" exitCode=0
Sep 11 14:20:17.232283 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.231953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n" event={"ID":"fa5cf026-ae2e-4204-a1be-b67217d18de8","Type":"ContainerDied","Data":"588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22"}
Sep 11 14:20:17.232450 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.232433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n" event={"ID":"fa5cf026-ae2e-4204-a1be-b67217d18de8","Type":"ContainerDied","Data":"3d969f91ed9f487fec4819a8417bcdfb1b97bfcd5ade2bb3d73c0bf7292973b9"}
Sep 11 14:20:17.232562 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.232549    2561 scope.go:117] "RemoveContainer" containerID="588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22"
Sep 11 14:20:17.232764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.231996    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"
Sep 11 14:20:17.255066 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.255044    2561 scope.go:117] "RemoveContainer" containerID="588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22"
Sep 11 14:20:17.255417 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:20:17.255397    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22\": container with ID starting with 588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22 not found: ID does not exist" containerID="588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22"
Sep 11 14:20:17.255481 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.255430    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22"} err="failed to get container status \"588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22\": rpc error: code = NotFound desc = could not find container \"588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22\": container with ID starting with 588cee6451b70c319a60de8742c44de09e325c6e7b514a91542a59f1a7b8da22 not found: ID does not exist"
Sep 11 14:20:17.287768 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.287742    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"]
Sep 11 14:20:17.294863 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:17.294842    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5cf565-9d59475bb-svs6n"]
Sep 11 14:20:18.117476 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:18.117438    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e95f6f-5c9e-42ba-8596-76a036f4638c" path="/var/lib/kubelet/pods/41e95f6f-5c9e-42ba-8596-76a036f4638c/volumes"
Sep 11 14:20:18.117880 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:18.117719    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965aa76f-39b8-4f84-a5fc-08e784a6fee7" path="/var/lib/kubelet/pods/965aa76f-39b8-4f84-a5fc-08e784a6fee7/volumes"
Sep 11 14:20:18.118026 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:18.118011    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5cf026-ae2e-4204-a1be-b67217d18de8" path="/var/lib/kubelet/pods/fa5cf026-ae2e-4204-a1be-b67217d18de8/volumes"
Sep 11 14:20:24.213921 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.213878    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:20:24.214367 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.214315    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:20:24.272215 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.272176    2561 generic.go:356] "Generic (PLEG): container finished" podID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerID="926f9818ff532ed0acc2d6605f4eee56393d698a871e84c46a31dd9087280586" exitCode=0
Sep 11 14:20:24.272372 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.272255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" event={"ID":"8bd10f49-bd76-441e-abf6-5b0d8c112275","Type":"ContainerDied","Data":"926f9818ff532ed0acc2d6605f4eee56393d698a871e84c46a31dd9087280586"}
Sep 11 14:20:24.272778 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.272762    2561 scope.go:117] "RemoveContainer" containerID="926f9818ff532ed0acc2d6605f4eee56393d698a871e84c46a31dd9087280586"
Sep 11 14:20:24.274013 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.273977    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:20:24.274111 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.274061    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:20:24.274111 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:24.274093    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 14:20:25.279166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:25.279132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" event={"ID":"8bd10f49-bd76-441e-abf6-5b0d8c112275","Type":"ContainerStarted","Data":"9baf75c29a44aad52ab46c44ace5e122b20c619f344fcfddc3a036f2dda4eb74"}
Sep 11 14:20:25.783412 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:25.783375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:25.783618 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:25.783596    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 14:20:25.783698 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:25.783651    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 14:20:26.783356 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:26.783316    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 14:20:26.783748 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:26.783385    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 14:20:27.783210 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:27.783166    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 14:20:27.783399 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:27.783230    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 14:20:27.911443 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:27.911405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:20:28.783420 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:28.783377    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 14:20:28.783838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:28.783439    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 14:20:29.783859 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:29.783817    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 14:20:29.784303 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:29.783888    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 14:20:30.808113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:30.808078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:30.808579 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:30.808445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:30.809636 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:30.809612    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:20:32.375699 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:32.375669    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j_4a0dfb4f-5789-44a6-8d5a-4cc0848a0376/payload-processing/0.log"
Sep 11 14:20:42.099675 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.099639    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"]
Sep 11 14:20:42.100124 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.100020    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5cf026-ae2e-4204-a1be-b67217d18de8" containerName="payload-pre-processing"
Sep 11 14:20:42.100124 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.100033    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5cf026-ae2e-4204-a1be-b67217d18de8" containerName="payload-pre-processing"
Sep 11 14:20:42.100124 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.100096    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5cf026-ae2e-4204-a1be-b67217d18de8" containerName="payload-pre-processing"
Sep 11 14:20:42.103091 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.103069    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.106362 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.106338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e6521d\"/\"default-dockercfg-ksq9z\""
Sep 11 14:20:42.107174 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.107154    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e6521d\"/\"kube-root-ca.crt\""
Sep 11 14:20:42.107446 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.107430    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e6521d\"/\"ipp-route-e6521d-kserve-self-signed-certs\""
Sep 11 14:20:42.107568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.107546    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e6521d\"/\"openshift-service-ca.crt\""
Sep 11 14:20:42.119559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.119532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"]
Sep 11 14:20:42.160181 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.160147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-home\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.160352 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.160190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c53164-e1f3-4097-9440-033a5a2ad246-tls-certs\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.160352 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.160207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8sjl\" (UniqueName: \"kubernetes.io/projected/60c53164-e1f3-4097-9440-033a5a2ad246-kube-api-access-x8sjl\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.160352 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.160302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-model-cache\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.160352 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.160349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-tmp-dir\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.160497 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.160388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-dshm\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261261 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-model-cache\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-tmp-dir\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-dshm\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-home\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261597 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c53164-e1f3-4097-9440-033a5a2ad246-tls-certs\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261597 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8sjl\" (UniqueName: \"kubernetes.io/projected/60c53164-e1f3-4097-9440-033a5a2ad246-kube-api-access-x8sjl\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261709 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-model-cache\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261709 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-tmp-dir\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.261838 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.261719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-home\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.263673 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.263649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-dshm\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.263921 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.263882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c53164-e1f3-4097-9440-033a5a2ad246-tls-certs\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.276556 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.276530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8sjl\" (UniqueName: \"kubernetes.io/projected/60c53164-e1f3-4097-9440-033a5a2ad246-kube-api-access-x8sjl\") pod \"ipp-route-e6521d-kserve-7776684f7f-n755n\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") " pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.414005 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.413898    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:42.690806 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:42.690753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"]
Sep 11 14:20:42.693022 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:20:42.692992    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60c53164_e1f3_4097_9440_033a5a2ad246.slice/crio-0b94f79bd5cb86c7bf5b07ad53f7ccea3928d40f0f1e11daea41f88ced011738 WatchSource:0}: Error finding container 0b94f79bd5cb86c7bf5b07ad53f7ccea3928d40f0f1e11daea41f88ced011738: Status 404 returned error can't find the container with id 0b94f79bd5cb86c7bf5b07ad53f7ccea3928d40f0f1e11daea41f88ced011738
Sep 11 14:20:43.364737 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:43.364699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n" event={"ID":"60c53164-e1f3-4097-9440-033a5a2ad246","Type":"ContainerStarted","Data":"90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105"}
Sep 11 14:20:43.364737 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:43.364743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n" event={"ID":"60c53164-e1f3-4097-9440-033a5a2ad246","Type":"ContainerStarted","Data":"0b94f79bd5cb86c7bf5b07ad53f7ccea3928d40f0f1e11daea41f88ced011738"}
Sep 11 14:20:43.386061 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:43.386011    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n" podStartSLOduration=1.385992766 podStartE2EDuration="1.385992766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:43.385141148 +0000 UTC m=+1691.752730591" watchObservedRunningTime="2026-09-11 14:20:43.385992766 +0000 UTC m=+1691.753582212"
Sep 11 14:20:46.196695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:46.196663    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:20:52.415094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:52.415058    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:52.415588 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:52.415107    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:52.427601 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:52.427580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:20:53.422112 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:20:53.422083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:21:01.899927 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:01.899836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:21:08.385528 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:08.385476    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j_4a0dfb4f-5789-44a6-8d5a-4cc0848a0376/payload-processing/0.log"
Sep 11 14:21:17.972923 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:17.972877    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"]
Sep 11 14:21:17.976407 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:17.976390    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:17.978979 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:17.978952    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-246a87\""
Sep 11 14:21:17.979107 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:17.979089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-246a87-dockercfg-k4tvs\""
Sep 11 14:21:17.987418 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:17.987392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"]
Sep 11 14:21:18.041294 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.041257    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"]
Sep 11 14:21:18.044575 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.044555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.058200 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.058177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"]
Sep 11 14:21:18.068739 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.068712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc0f02d-3530-4047-92a3-3de9d636d97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.068891 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.068745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccfc\" (UniqueName: \"kubernetes.io/projected/6bc0f02d-3530-4047-92a3-3de9d636d97a-kube-api-access-rccfc\") pod \"payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.169999 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.169959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2100ad5-18af-45ee-8be0-32a40de53e90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.169999 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.170014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcm2r\" (UniqueName: \"kubernetes.io/projected/f2100ad5-18af-45ee-8be0-32a40de53e90-kube-api-access-vcm2r\") pod \"payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.170232 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.170039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc0f02d-3530-4047-92a3-3de9d636d97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.170232 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.170066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccfc\" (UniqueName: \"kubernetes.io/projected/6bc0f02d-3530-4047-92a3-3de9d636d97a-kube-api-access-rccfc\") pod \"payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.170737 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.170715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc0f02d-3530-4047-92a3-3de9d636d97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.179886 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.179855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccfc\" (UniqueName: \"kubernetes.io/projected/6bc0f02d-3530-4047-92a3-3de9d636d97a-kube-api-access-rccfc\") pod \"payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.271117 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.271017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2100ad5-18af-45ee-8be0-32a40de53e90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.271117 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.271070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcm2r\" (UniqueName: \"kubernetes.io/projected/f2100ad5-18af-45ee-8be0-32a40de53e90-kube-api-access-vcm2r\") pod \"payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.271755 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.271735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2100ad5-18af-45ee-8be0-32a40de53e90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.281022 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.280998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcm2r\" (UniqueName: \"kubernetes.io/projected/f2100ad5-18af-45ee-8be0-32a40de53e90-kube-api-access-vcm2r\") pod \"payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.286816 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.286791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:18.355190 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.355157    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:18.560531 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.560480    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"]
Sep 11 14:21:18.563386 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:21:18.563360    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc0f02d_3530_4047_92a3_3de9d636d97a.slice/crio-bca4076ede8c5a13c256311eafb91f6b540c69d6933955999d632261007b0ddd WatchSource:0}: Error finding container bca4076ede8c5a13c256311eafb91f6b540c69d6933955999d632261007b0ddd: Status 404 returned error can't find the container with id bca4076ede8c5a13c256311eafb91f6b540c69d6933955999d632261007b0ddd
Sep 11 14:21:18.633888 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:18.633860    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"]
Sep 11 14:21:18.635958 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:21:18.635928    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2100ad5_18af_45ee_8be0_32a40de53e90.slice/crio-c898d9c71d38bcb1b8daf6ed5f382387c4da3bb09c7a390a84e2c12b927639de WatchSource:0}: Error finding container c898d9c71d38bcb1b8daf6ed5f382387c4da3bb09c7a390a84e2c12b927639de: Status 404 returned error can't find the container with id c898d9c71d38bcb1b8daf6ed5f382387c4da3bb09c7a390a84e2c12b927639de
Sep 11 14:21:19.538965 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.538881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd" event={"ID":"f2100ad5-18af-45ee-8be0-32a40de53e90","Type":"ContainerStarted","Data":"5d0f06c4ea56a332f2e3cd0cd1266992f30fe1b6da0cd8ec67304f3f52ef4b71"}
Sep 11 14:21:19.539478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.539006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd" event={"ID":"f2100ad5-18af-45ee-8be0-32a40de53e90","Type":"ContainerStarted","Data":"c898d9c71d38bcb1b8daf6ed5f382387c4da3bb09c7a390a84e2c12b927639de"}
Sep 11 14:21:19.539478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.539038    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:19.540852 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.540818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt" event={"ID":"6bc0f02d-3530-4047-92a3-3de9d636d97a","Type":"ContainerStarted","Data":"2a0a9c0de0b93d9c7bc0a95eff87cddff01eddc20b1577c8688c82b14edb359d"}
Sep 11 14:21:19.541014 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.540873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt" event={"ID":"6bc0f02d-3530-4047-92a3-3de9d636d97a","Type":"ContainerStarted","Data":"bca4076ede8c5a13c256311eafb91f6b540c69d6933955999d632261007b0ddd"}
Sep 11 14:21:19.541014 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.540944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:19.565420 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.565367    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd" podStartSLOduration=1.56535062 podStartE2EDuration="1.56535062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:19.564770632 +0000 UTC m=+1727.932360077" watchObservedRunningTime="2026-09-11 14:21:19.56535062 +0000 UTC m=+1727.932940063"
Sep 11 14:21:19.586691 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:19.586635    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt" podStartSLOduration=2.586592733 podStartE2EDuration="2.586592733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:19.585264343 +0000 UTC m=+1727.952853785" watchObservedRunningTime="2026-09-11 14:21:19.586592733 +0000 UTC m=+1727.954182178"
Sep 11 14:21:21.906205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:21.906167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:21:30.548542 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:30.548510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:30.549019 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:30.548582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:32.416199 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:32.416162    2561 scope.go:117] "RemoveContainer" containerID="b694821d851d63a15d7fa41d00eee80322bc761d07b1b135d5ebe8c469648e98"
Sep 11 14:21:54.528590 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.528553    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"]
Sep 11 14:21:54.529137 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.528793    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd" podUID="f2100ad5-18af-45ee-8be0-32a40de53e90" containerName="payload-processing" containerID="cri-o://5d0f06c4ea56a332f2e3cd0cd1266992f30fe1b6da0cd8ec67304f3f52ef4b71" gracePeriod=30
Sep 11 14:21:54.558088 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.558055    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"]
Sep 11 14:21:54.558573 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.558534    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt" podUID="6bc0f02d-3530-4047-92a3-3de9d636d97a" containerName="payload-pre-processing" containerID="cri-o://2a0a9c0de0b93d9c7bc0a95eff87cddff01eddc20b1577c8688c82b14edb359d" gracePeriod=30
Sep 11 14:21:54.711788 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.711747    2561 generic.go:356] "Generic (PLEG): container finished" podID="6bc0f02d-3530-4047-92a3-3de9d636d97a" containerID="2a0a9c0de0b93d9c7bc0a95eff87cddff01eddc20b1577c8688c82b14edb359d" exitCode=0
Sep 11 14:21:54.711983 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.711843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt" event={"ID":"6bc0f02d-3530-4047-92a3-3de9d636d97a","Type":"ContainerDied","Data":"2a0a9c0de0b93d9c7bc0a95eff87cddff01eddc20b1577c8688c82b14edb359d"}
Sep 11 14:21:54.713833 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.713806    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2100ad5-18af-45ee-8be0-32a40de53e90" containerID="5d0f06c4ea56a332f2e3cd0cd1266992f30fe1b6da0cd8ec67304f3f52ef4b71" exitCode=0
Sep 11 14:21:54.713983 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.713861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd" event={"ID":"f2100ad5-18af-45ee-8be0-32a40de53e90","Type":"ContainerDied","Data":"5d0f06c4ea56a332f2e3cd0cd1266992f30fe1b6da0cd8ec67304f3f52ef4b71"}
Sep 11 14:21:54.827139 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.827116    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:54.829640 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.829619    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:54.895763 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.895731    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc0f02d-3530-4047-92a3-3de9d636d97a-plugins-config-volume\") pod \"6bc0f02d-3530-4047-92a3-3de9d636d97a\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") "
Sep 11 14:21:54.895944 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.895809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rccfc\" (UniqueName: \"kubernetes.io/projected/6bc0f02d-3530-4047-92a3-3de9d636d97a-kube-api-access-rccfc\") pod \"6bc0f02d-3530-4047-92a3-3de9d636d97a\" (UID: \"6bc0f02d-3530-4047-92a3-3de9d636d97a\") "
Sep 11 14:21:54.895944 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.895827    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcm2r\" (UniqueName: \"kubernetes.io/projected/f2100ad5-18af-45ee-8be0-32a40de53e90-kube-api-access-vcm2r\") pod \"f2100ad5-18af-45ee-8be0-32a40de53e90\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") "
Sep 11 14:21:54.896050 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.895946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2100ad5-18af-45ee-8be0-32a40de53e90-plugins-config-volume\") pod \"f2100ad5-18af-45ee-8be0-32a40de53e90\" (UID: \"f2100ad5-18af-45ee-8be0-32a40de53e90\") "
Sep 11 14:21:54.896311 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.896277    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc0f02d-3530-4047-92a3-3de9d636d97a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bc0f02d-3530-4047-92a3-3de9d636d97a" (UID: "6bc0f02d-3530-4047-92a3-3de9d636d97a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:21:54.896311 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.896297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2100ad5-18af-45ee-8be0-32a40de53e90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2100ad5-18af-45ee-8be0-32a40de53e90" (UID: "f2100ad5-18af-45ee-8be0-32a40de53e90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:21:54.898526 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.898500    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2100ad5-18af-45ee-8be0-32a40de53e90-kube-api-access-vcm2r" (OuterVolumeSpecName: "kube-api-access-vcm2r") pod "f2100ad5-18af-45ee-8be0-32a40de53e90" (UID: "f2100ad5-18af-45ee-8be0-32a40de53e90"). InnerVolumeSpecName "kube-api-access-vcm2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:21:54.898610 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.898534    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc0f02d-3530-4047-92a3-3de9d636d97a-kube-api-access-rccfc" (OuterVolumeSpecName: "kube-api-access-rccfc") pod "6bc0f02d-3530-4047-92a3-3de9d636d97a" (UID: "6bc0f02d-3530-4047-92a3-3de9d636d97a"). InnerVolumeSpecName "kube-api-access-rccfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:21:54.997593 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.997558    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc0f02d-3530-4047-92a3-3de9d636d97a-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:21:54.997593 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.997588    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rccfc\" (UniqueName: \"kubernetes.io/projected/6bc0f02d-3530-4047-92a3-3de9d636d97a-kube-api-access-rccfc\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:21:54.997593 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.997599    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcm2r\" (UniqueName: \"kubernetes.io/projected/f2100ad5-18af-45ee-8be0-32a40de53e90-kube-api-access-vcm2r\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:21:54.997828 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:54.997607    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2100ad5-18af-45ee-8be0-32a40de53e90-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:21:55.725516 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.725481    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"
Sep 11 14:21:55.725996 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.725487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd" event={"ID":"f2100ad5-18af-45ee-8be0-32a40de53e90","Type":"ContainerDied","Data":"c898d9c71d38bcb1b8daf6ed5f382387c4da3bb09c7a390a84e2c12b927639de"}
Sep 11 14:21:55.725996 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.725658    2561 scope.go:117] "RemoveContainer" containerID="5d0f06c4ea56a332f2e3cd0cd1266992f30fe1b6da0cd8ec67304f3f52ef4b71"
Sep 11 14:21:55.727130 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.726986    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt" event={"ID":"6bc0f02d-3530-4047-92a3-3de9d636d97a","Type":"ContainerDied","Data":"bca4076ede8c5a13c256311eafb91f6b540c69d6933955999d632261007b0ddd"}
Sep 11 14:21:55.727130 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.727028    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"
Sep 11 14:21:55.735440 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.735411    2561 scope.go:117] "RemoveContainer" containerID="2a0a9c0de0b93d9c7bc0a95eff87cddff01eddc20b1577c8688c82b14edb359d"
Sep 11 14:21:55.767171 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.767146    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"]
Sep 11 14:21:55.784174 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.784152    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-246a87-574dd9bf6c-l4htd"]
Sep 11 14:21:55.812055 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.812026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"]
Sep 11 14:21:55.821824 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:55.821798    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-246a87-549c67948b-d25bt"]
Sep 11 14:21:56.117308 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:56.117271    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc0f02d-3530-4047-92a3-3de9d636d97a" path="/var/lib/kubelet/pods/6bc0f02d-3530-4047-92a3-3de9d636d97a/volumes"
Sep 11 14:21:56.117712 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:56.117692    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2100ad5-18af-45ee-8be0-32a40de53e90" path="/var/lib/kubelet/pods/f2100ad5-18af-45ee-8be0-32a40de53e90/volumes"
Sep 11 14:21:59.411797 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:21:59.411759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:22:19.707341 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:19.707306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:22:21.983248 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:21.983216    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"]
Sep 11 14:22:21.983666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:21.983443    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" podUID="20c798c7-5445-4cb2-88e4-2b25478bd892" containerName="payload-pre-processing" containerID="cri-o://cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b" gracePeriod=30
Sep 11 14:22:22.091626 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.091587    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" podUID="20c798c7-5445-4cb2-88e4-2b25478bd892" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.75:9004: connect: connection refused"
Sep 11 14:22:22.237542 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.237477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:22:22.341078 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.341045    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhfcn\" (UniqueName: \"kubernetes.io/projected/20c798c7-5445-4cb2-88e4-2b25478bd892-kube-api-access-vhfcn\") pod \"20c798c7-5445-4cb2-88e4-2b25478bd892\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") "
Sep 11 14:22:22.341273 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.341135    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c798c7-5445-4cb2-88e4-2b25478bd892-plugins-config-volume\") pod \"20c798c7-5445-4cb2-88e4-2b25478bd892\" (UID: \"20c798c7-5445-4cb2-88e4-2b25478bd892\") "
Sep 11 14:22:22.341532 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.341509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c798c7-5445-4cb2-88e4-2b25478bd892-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20c798c7-5445-4cb2-88e4-2b25478bd892" (UID: "20c798c7-5445-4cb2-88e4-2b25478bd892"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:22.343274 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.343254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c798c7-5445-4cb2-88e4-2b25478bd892-kube-api-access-vhfcn" (OuterVolumeSpecName: "kube-api-access-vhfcn") pod "20c798c7-5445-4cb2-88e4-2b25478bd892" (UID: "20c798c7-5445-4cb2-88e4-2b25478bd892"). InnerVolumeSpecName "kube-api-access-vhfcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:22:22.442365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.442328    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c798c7-5445-4cb2-88e4-2b25478bd892-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:22.442365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.442353    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhfcn\" (UniqueName: \"kubernetes.io/projected/20c798c7-5445-4cb2-88e4-2b25478bd892-kube-api-access-vhfcn\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:22.855011 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.854975    2561 generic.go:356] "Generic (PLEG): container finished" podID="20c798c7-5445-4cb2-88e4-2b25478bd892" containerID="cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b" exitCode=0
Sep 11 14:22:22.855206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.855017    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"
Sep 11 14:22:22.855206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.855027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" event={"ID":"20c798c7-5445-4cb2-88e4-2b25478bd892","Type":"ContainerDied","Data":"cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b"}
Sep 11 14:22:22.855206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.855103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr" event={"ID":"20c798c7-5445-4cb2-88e4-2b25478bd892","Type":"ContainerDied","Data":"cffe09598dfc5c6decfa15ef25753ba31490beef392b5f2dde568b8beef621da"}
Sep 11 14:22:22.855206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.855125    2561 scope.go:117] "RemoveContainer" containerID="cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b"
Sep 11 14:22:22.863793 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.863774    2561 scope.go:117] "RemoveContainer" containerID="cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b"
Sep 11 14:22:22.864133 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:22:22.864116    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b\": container with ID starting with cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b not found: ID does not exist" containerID="cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b"
Sep 11 14:22:22.864224 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.864137    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b"} err="failed to get container status \"cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b\": rpc error: code = NotFound desc = could not find container \"cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b\": container with ID starting with cd01bb6e559c95976f18bcd151e0ac6e577ef33b9dbce394647003ede706835b not found: ID does not exist"
Sep 11 14:22:22.879284 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.879258    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"]
Sep 11 14:22:22.886743 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:22.886720    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6521d-66b4b7bd46-5rmfr"]
Sep 11 14:22:24.116500 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:24.116462    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c798c7-5445-4cb2-88e4-2b25478bd892" path="/var/lib/kubelet/pods/20c798c7-5445-4cb2-88e4-2b25478bd892/volumes"
Sep 11 14:22:27.591217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:27.591180    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"]
Sep 11 14:22:32.208264 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:32.208234    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:22:32.213013 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:32.212987    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:22:32.698179 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:32.698146    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:22:42.609927 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.609874    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n" podUID="60c53164-e1f3-4097-9440-033a5a2ad246" containerName="main" containerID="cri-o://90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105" gracePeriod=45
Sep 11 14:22:42.862982 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.862921    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:22:42.907193 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-tmp-dir\") pod \"60c53164-e1f3-4097-9440-033a5a2ad246\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") "
Sep 11 14:22:42.907193 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-dshm\") pod \"60c53164-e1f3-4097-9440-033a5a2ad246\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") "
Sep 11 14:22:42.907445 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c53164-e1f3-4097-9440-033a5a2ad246-tls-certs\") pod \"60c53164-e1f3-4097-9440-033a5a2ad246\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") "
Sep 11 14:22:42.907445 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-home\") pod \"60c53164-e1f3-4097-9440-033a5a2ad246\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") "
Sep 11 14:22:42.907568 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907473    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60c53164-e1f3-4097-9440-033a5a2ad246" (UID: "60c53164-e1f3-4097-9440-033a5a2ad246"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:42.907805 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907738    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-home" (OuterVolumeSpecName: "home") pod "60c53164-e1f3-4097-9440-033a5a2ad246" (UID: "60c53164-e1f3-4097-9440-033a5a2ad246"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:42.907805 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907779    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-model-cache\") pod \"60c53164-e1f3-4097-9440-033a5a2ad246\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") "
Sep 11 14:22:42.908011 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.907826    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8sjl\" (UniqueName: \"kubernetes.io/projected/60c53164-e1f3-4097-9440-033a5a2ad246-kube-api-access-x8sjl\") pod \"60c53164-e1f3-4097-9440-033a5a2ad246\" (UID: \"60c53164-e1f3-4097-9440-033a5a2ad246\") "
Sep 11 14:22:42.908079 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.908037    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-model-cache" (OuterVolumeSpecName: "model-cache") pod "60c53164-e1f3-4097-9440-033a5a2ad246" (UID: "60c53164-e1f3-4097-9440-033a5a2ad246"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:42.908153 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.908130    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:42.908214 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.908158    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:42.908214 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.908171    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:42.909904 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.909875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c53164-e1f3-4097-9440-033a5a2ad246-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60c53164-e1f3-4097-9440-033a5a2ad246" (UID: "60c53164-e1f3-4097-9440-033a5a2ad246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:22:42.909904 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.909901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-dshm" (OuterVolumeSpecName: "dshm") pod "60c53164-e1f3-4097-9440-033a5a2ad246" (UID: "60c53164-e1f3-4097-9440-033a5a2ad246"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:22:42.910109 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.910036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c53164-e1f3-4097-9440-033a5a2ad246-kube-api-access-x8sjl" (OuterVolumeSpecName: "kube-api-access-x8sjl") pod "60c53164-e1f3-4097-9440-033a5a2ad246" (UID: "60c53164-e1f3-4097-9440-033a5a2ad246"). InnerVolumeSpecName "kube-api-access-x8sjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:22:42.945618 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.945588    2561 generic.go:356] "Generic (PLEG): container finished" podID="60c53164-e1f3-4097-9440-033a5a2ad246" containerID="90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105" exitCode=0
Sep 11 14:22:42.945776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.945649    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"
Sep 11 14:22:42.945776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.945662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n" event={"ID":"60c53164-e1f3-4097-9440-033a5a2ad246","Type":"ContainerDied","Data":"90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105"}
Sep 11 14:22:42.945776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.945693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n" event={"ID":"60c53164-e1f3-4097-9440-033a5a2ad246","Type":"ContainerDied","Data":"0b94f79bd5cb86c7bf5b07ad53f7ccea3928d40f0f1e11daea41f88ced011738"}
Sep 11 14:22:42.945776 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.945710    2561 scope.go:117] "RemoveContainer" containerID="90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105"
Sep 11 14:22:42.954282 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.954263    2561 scope.go:117] "RemoveContainer" containerID="90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105"
Sep 11 14:22:42.954545 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:22:42.954523    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105\": container with ID starting with 90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105 not found: ID does not exist" containerID="90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105"
Sep 11 14:22:42.954622 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.954552    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105"} err="failed to get container status \"90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105\": rpc error: code = NotFound desc = could not find container \"90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105\": container with ID starting with 90f135bbbd7ed4b55a8ca8ac65c28b38187ef9f776229da7eaaadf949018e105 not found: ID does not exist"
Sep 11 14:22:42.972758 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.972684    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"]
Sep 11 14:22:42.982748 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:42.982723    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e6521d/ipp-route-e6521d-kserve-7776684f7f-n755n"]
Sep 11 14:22:43.008686 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:43.008657    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c53164-e1f3-4097-9440-033a5a2ad246-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.008686 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:43.008683    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c53164-e1f3-4097-9440-033a5a2ad246-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.008850 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:43.008698    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8sjl\" (UniqueName: \"kubernetes.io/projected/60c53164-e1f3-4097-9440-033a5a2ad246-kube-api-access-x8sjl\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:22:44.117081 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:22:44.117048    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c53164-e1f3-4097-9440-033a5a2ad246" path="/var/lib/kubelet/pods/60c53164-e1f3-4097-9440-033a5a2ad246/volumes"
Sep 11 14:23:23.365580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.365530    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"]
Sep 11 14:23:23.366377 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.366314    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j" podUID="4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" containerName="payload-processing" containerID="cri-o://54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95" gracePeriod=30
Sep 11 14:23:23.382206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.382161    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"]
Sep 11 14:23:23.382456 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.382428    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc" podUID="f0b1b244-87d4-4164-9a1a-2dc3c8941408" containerName="payload-pre-processing" containerID="cri-o://f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda" gracePeriod=30
Sep 11 14:23:23.653558 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.653532    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:23:23.656234 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.656209    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:23:23.753997 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.753963    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t725b\" (UniqueName: \"kubernetes.io/projected/f0b1b244-87d4-4164-9a1a-2dc3c8941408-kube-api-access-t725b\") pod \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") "
Sep 11 14:23:23.754205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.754005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x6k2\" (UniqueName: \"kubernetes.io/projected/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-kube-api-access-9x6k2\") pod \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") "
Sep 11 14:23:23.754205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.754038    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1b244-87d4-4164-9a1a-2dc3c8941408-plugins-config-volume\") pod \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\" (UID: \"f0b1b244-87d4-4164-9a1a-2dc3c8941408\") "
Sep 11 14:23:23.754205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.754085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-plugins-config-volume\") pod \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\" (UID: \"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376\") "
Sep 11 14:23:23.754506 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.754475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b1b244-87d4-4164-9a1a-2dc3c8941408-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0b1b244-87d4-4164-9a1a-2dc3c8941408" (UID: "f0b1b244-87d4-4164-9a1a-2dc3c8941408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:23:23.754623 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.754542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" (UID: "4a0dfb4f-5789-44a6-8d5a-4cc0848a0376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:23:23.756251 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.756220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-kube-api-access-9x6k2" (OuterVolumeSpecName: "kube-api-access-9x6k2") pod "4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" (UID: "4a0dfb4f-5789-44a6-8d5a-4cc0848a0376"). InnerVolumeSpecName "kube-api-access-9x6k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:23:23.756383 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.756358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b1b244-87d4-4164-9a1a-2dc3c8941408-kube-api-access-t725b" (OuterVolumeSpecName: "kube-api-access-t725b") pod "f0b1b244-87d4-4164-9a1a-2dc3c8941408" (UID: "f0b1b244-87d4-4164-9a1a-2dc3c8941408"). InnerVolumeSpecName "kube-api-access-t725b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:23:23.854684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.854646    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:23.854684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.854675    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t725b\" (UniqueName: \"kubernetes.io/projected/f0b1b244-87d4-4164-9a1a-2dc3c8941408-kube-api-access-t725b\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:23.854684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.854685    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6k2\" (UniqueName: \"kubernetes.io/projected/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376-kube-api-access-9x6k2\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:23.854684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:23.854694    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b1b244-87d4-4164-9a1a-2dc3c8941408-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:24.133106 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.133077    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" containerID="54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95" exitCode=0
Sep 11 14:23:24.133219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.133115    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"
Sep 11 14:23:24.133219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.133155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j" event={"ID":"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376","Type":"ContainerDied","Data":"54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95"}
Sep 11 14:23:24.133219 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.133215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j" event={"ID":"4a0dfb4f-5789-44a6-8d5a-4cc0848a0376","Type":"ContainerDied","Data":"25ab8278bdf0ac7ccb4b65958072f43816b53d307b738740d38bdb2fa9995586"}
Sep 11 14:23:24.133366 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.133241    2561 scope.go:117] "RemoveContainer" containerID="54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95"
Sep 11 14:23:24.134424 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.134393    2561 generic.go:356] "Generic (PLEG): container finished" podID="f0b1b244-87d4-4164-9a1a-2dc3c8941408" containerID="f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda" exitCode=0
Sep 11 14:23:24.134538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.134445    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"
Sep 11 14:23:24.134538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.134475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc" event={"ID":"f0b1b244-87d4-4164-9a1a-2dc3c8941408","Type":"ContainerDied","Data":"f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda"}
Sep 11 14:23:24.134538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.134515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc" event={"ID":"f0b1b244-87d4-4164-9a1a-2dc3c8941408","Type":"ContainerDied","Data":"2ef08dbbe272f671a544015718a9bae4851d51c5a6eed12c3c7b085a963e6972"}
Sep 11 14:23:24.141857 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.141844    2561 scope.go:117] "RemoveContainer" containerID="54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95"
Sep 11 14:23:24.142174 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:23:24.142155    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95\": container with ID starting with 54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95 not found: ID does not exist" containerID="54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95"
Sep 11 14:23:24.142229 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.142186    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95"} err="failed to get container status \"54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95\": rpc error: code = NotFound desc = could not find container \"54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95\": container with ID starting with 54019d6d8f7ed7061d80f4732f7571a30c9e73cb66d8607227a3118e34e47a95 not found: ID does not exist"
Sep 11 14:23:24.142229 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.142202    2561 scope.go:117] "RemoveContainer" containerID="f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda"
Sep 11 14:23:24.151484 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.151433    2561 scope.go:117] "RemoveContainer" containerID="f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda"
Sep 11 14:23:24.151998 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:23:24.151970    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda\": container with ID starting with f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda not found: ID does not exist" containerID="f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda"
Sep 11 14:23:24.152111 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.152015    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda"} err="failed to get container status \"f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda\": rpc error: code = NotFound desc = could not find container \"f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda\": container with ID starting with f32bed8fa7d69d0944a933d50b038f3e216a3bc19e903754b0b908017b1aeeda not found: ID does not exist"
Sep 11 14:23:24.153357 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.153334    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"]
Sep 11 14:23:24.155192 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.155174    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d9cd47-866dd74b9c-8m65j"]
Sep 11 14:23:24.166623 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.166600    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"]
Sep 11 14:23:24.170373 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:24.170352    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d9cd47-8dcc487f8-knbfc"]
Sep 11 14:23:26.116876 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:26.116841    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" path="/var/lib/kubelet/pods/4a0dfb4f-5789-44a6-8d5a-4cc0848a0376/volumes"
Sep 11 14:23:26.117291 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:26.117233    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b1b244-87d4-4164-9a1a-2dc3c8941408" path="/var/lib/kubelet/pods/f0b1b244-87d4-4164-9a1a-2dc3c8941408/volumes"
Sep 11 14:23:26.510634 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:26.510547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:23:31.068637 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:31.068599    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"]
Sep 11 14:23:31.069161 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:31.068887    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy" containerID="cri-o://9baf75c29a44aad52ab46c44ace5e122b20c619f344fcfddc3a036f2dda4eb74" gracePeriod=30
Sep 11 14:23:36.191748 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.191711    2561 generic.go:356] "Generic (PLEG): container finished" podID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerID="9baf75c29a44aad52ab46c44ace5e122b20c619f344fcfddc3a036f2dda4eb74" exitCode=0
Sep 11 14:23:36.191748 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.191739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" event={"ID":"8bd10f49-bd76-441e-abf6-5b0d8c112275","Type":"ContainerDied","Data":"9baf75c29a44aad52ab46c44ace5e122b20c619f344fcfddc3a036f2dda4eb74"}
Sep 11 14:23:36.192189 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.191804    2561 scope.go:117] "RemoveContainer" containerID="926f9818ff532ed0acc2d6605f4eee56393d698a871e84c46a31dd9087280586"
Sep 11 14:23:36.312125 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.312097    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:23:36.365152 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-token\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365152 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365114    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-data\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365152 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-credential-socket\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365163    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-envoy\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-socket\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365237    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-podinfo\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365272    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbb8k\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-kube-api-access-kbb8k\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd10f49-bd76-441e-abf6-5b0d8c112275-istiod-ca-cert\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-certs\") pod \"8bd10f49-bd76-441e-abf6-5b0d8c112275\" (UID: \"8bd10f49-bd76-441e-abf6-5b0d8c112275\") "
Sep 11 14:23:36.365458 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365401    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:23:36.365860 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:23:36.365860 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:23:36.365860 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:23:36.365860 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365668    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-data\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.365860 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.365684    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-credential-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.366160 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.366066    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd10f49-bd76-441e-abf6-5b0d8c112275-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:23:36.367631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.367601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:23:36.367940 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.367885    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:23:36.368053 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.368000    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-kube-api-access-kbb8k" (OuterVolumeSpecName: "kube-api-access-kbb8k") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "kube-api-access-kbb8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:23:36.368131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.368104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bd10f49-bd76-441e-abf6-5b0d8c112275" (UID: "8bd10f49-bd76-441e-abf6-5b0d8c112275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:23:36.467020 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.466982    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-envoy\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.467020 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.467012    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-socket\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.467020 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.467021    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-podinfo\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.467325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.467032    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbb8k\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-kube-api-access-kbb8k\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.467325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.467047    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd10f49-bd76-441e-abf6-5b0d8c112275-istiod-ca-cert\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.467325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.467059    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd10f49-bd76-441e-abf6-5b0d8c112275-workload-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:36.467325 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:36.467070    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd10f49-bd76-441e-abf6-5b0d8c112275-istio-token\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:37.197052 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:37.197016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9" event={"ID":"8bd10f49-bd76-441e-abf6-5b0d8c112275","Type":"ContainerDied","Data":"dd0fda41c3bdc34f95de99b5a47f5b2d5d16d85eae245fbf2557b6e825939c56"}
Sep 11 14:23:37.197539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:37.197080    2561 scope.go:117] "RemoveContainer" containerID="9baf75c29a44aad52ab46c44ace5e122b20c619f344fcfddc3a036f2dda4eb74"
Sep 11 14:23:37.197539 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:37.197032    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"
Sep 11 14:23:37.222581 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:37.222552    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"]
Sep 11 14:23:37.227365 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:37.227340    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d9cd47-openshift-default-746df77886-btzh9"]
Sep 11 14:23:38.116693 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:38.116658    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" path="/var/lib/kubelet/pods/8bd10f49-bd76-441e-abf6-5b0d8c112275/volumes"
Sep 11 14:23:42.174317 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.174283    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"]
Sep 11 14:23:42.174832 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.174485    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg" podUID="8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" containerName="payload-processing" containerID="cri-o://34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb" gracePeriod=30
Sep 11 14:23:42.184811 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.184777    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"]
Sep 11 14:23:42.185064 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.185024    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx" podUID="60b91c86-18ff-4eea-ba02-eefc1294426d" containerName="payload-pre-processing" containerID="cri-o://15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6" gracePeriod=30
Sep 11 14:23:42.453633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.453604    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:23:42.455623 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.455606    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:23:42.521532 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.521491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-plugins-config-volume\") pod \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") "
Sep 11 14:23:42.521728 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.521561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2trxk\" (UniqueName: \"kubernetes.io/projected/60b91c86-18ff-4eea-ba02-eefc1294426d-kube-api-access-2trxk\") pod \"60b91c86-18ff-4eea-ba02-eefc1294426d\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") "
Sep 11 14:23:42.521728 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.521628    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b91c86-18ff-4eea-ba02-eefc1294426d-plugins-config-volume\") pod \"60b91c86-18ff-4eea-ba02-eefc1294426d\" (UID: \"60b91c86-18ff-4eea-ba02-eefc1294426d\") "
Sep 11 14:23:42.521728 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.521650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5qdq\" (UniqueName: \"kubernetes.io/projected/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-kube-api-access-z5qdq\") pod \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\" (UID: \"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e\") "
Sep 11 14:23:42.522014 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.521981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" (UID: "8dfe6d6b-49f4-428a-b22d-5f5d36a9363e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:23:42.522014 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.521993    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b91c86-18ff-4eea-ba02-eefc1294426d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60b91c86-18ff-4eea-ba02-eefc1294426d" (UID: "60b91c86-18ff-4eea-ba02-eefc1294426d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:23:42.523714 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.523689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-kube-api-access-z5qdq" (OuterVolumeSpecName: "kube-api-access-z5qdq") pod "8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" (UID: "8dfe6d6b-49f4-428a-b22d-5f5d36a9363e"). InnerVolumeSpecName "kube-api-access-z5qdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:23:42.523795 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.523723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b91c86-18ff-4eea-ba02-eefc1294426d-kube-api-access-2trxk" (OuterVolumeSpecName: "kube-api-access-2trxk") pod "60b91c86-18ff-4eea-ba02-eefc1294426d" (UID: "60b91c86-18ff-4eea-ba02-eefc1294426d"). InnerVolumeSpecName "kube-api-access-2trxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:23:42.622802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.622759    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trxk\" (UniqueName: \"kubernetes.io/projected/60b91c86-18ff-4eea-ba02-eefc1294426d-kube-api-access-2trxk\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:42.622802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.622793    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b91c86-18ff-4eea-ba02-eefc1294426d-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:42.622802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.622807    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qdq\" (UniqueName: \"kubernetes.io/projected/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-kube-api-access-z5qdq\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:42.623083 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:42.622818    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:43.227176 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.227142    2561 generic.go:356] "Generic (PLEG): container finished" podID="8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" containerID="34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb" exitCode=0
Sep 11 14:23:43.227634 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.227184    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"
Sep 11 14:23:43.227634 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.227229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg" event={"ID":"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e","Type":"ContainerDied","Data":"34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb"}
Sep 11 14:23:43.227634 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.227280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg" event={"ID":"8dfe6d6b-49f4-428a-b22d-5f5d36a9363e","Type":"ContainerDied","Data":"297528b67d5fb5ec6c092e8dcdc256d85c6d70c53fad5bca046678997d13d1c2"}
Sep 11 14:23:43.227634 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.227308    2561 scope.go:117] "RemoveContainer" containerID="34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb"
Sep 11 14:23:43.228769 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.228746    2561 generic.go:356] "Generic (PLEG): container finished" podID="60b91c86-18ff-4eea-ba02-eefc1294426d" containerID="15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6" exitCode=0
Sep 11 14:23:43.228861 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.228784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx" event={"ID":"60b91c86-18ff-4eea-ba02-eefc1294426d","Type":"ContainerDied","Data":"15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6"}
Sep 11 14:23:43.228861 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.228801    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"
Sep 11 14:23:43.228861 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.228839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx" event={"ID":"60b91c86-18ff-4eea-ba02-eefc1294426d","Type":"ContainerDied","Data":"49e599875fbe62d2b7d32b21dd9a2962bac37b8b9c0230aba377fde4de5fb74b"}
Sep 11 14:23:43.236874 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.236857    2561 scope.go:117] "RemoveContainer" containerID="34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb"
Sep 11 14:23:43.237275 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:23:43.237256    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb\": container with ID starting with 34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb not found: ID does not exist" containerID="34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb"
Sep 11 14:23:43.237374 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.237286    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb"} err="failed to get container status \"34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb\": rpc error: code = NotFound desc = could not find container \"34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb\": container with ID starting with 34baac8ff91ed78a98a00c9f7476434bfd2922b01d1c2300dc735a9e29862adb not found: ID does not exist"
Sep 11 14:23:43.237374 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.237309    2561 scope.go:117] "RemoveContainer" containerID="15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6"
Sep 11 14:23:43.245062 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.245043    2561 scope.go:117] "RemoveContainer" containerID="15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6"
Sep 11 14:23:43.245344 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:23:43.245325    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6\": container with ID starting with 15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6 not found: ID does not exist" containerID="15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6"
Sep 11 14:23:43.245425 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.245352    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6"} err="failed to get container status \"15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6\": rpc error: code = NotFound desc = could not find container \"15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6\": container with ID starting with 15af2b39d901b5e432f42bc52b8e606329fe460ddb9634c19a68dfc42bac17c6 not found: ID does not exist"
Sep 11 14:23:43.271128 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.271093    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"]
Sep 11 14:23:43.274709 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.274675    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb65d4-75c688654c-x7nlg"]
Sep 11 14:23:43.298399 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.298360    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"]
Sep 11 14:23:43.304321 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:43.304295    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb65d4-548d457c66-kxfbx"]
Sep 11 14:23:44.117535 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:44.117492    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b91c86-18ff-4eea-ba02-eefc1294426d" path="/var/lib/kubelet/pods/60b91c86-18ff-4eea-ba02-eefc1294426d/volumes"
Sep 11 14:23:44.117950 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:44.117931    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" path="/var/lib/kubelet/pods/8dfe6d6b-49f4-428a-b22d-5f5d36a9363e/volumes"
Sep 11 14:23:45.024898 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:45.024864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:23:50.020254 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020219    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"]
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020488    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" containerName="payload-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020498    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" containerName="payload-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020509    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c798c7-5445-4cb2-88e4-2b25478bd892" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c798c7-5445-4cb2-88e4-2b25478bd892" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020527    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" containerName="payload-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020534    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" containerName="payload-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020541    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020546    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020552    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c53164-e1f3-4097-9440-033a5a2ad246" containerName="main"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020557    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c53164-e1f3-4097-9440-033a5a2ad246" containerName="main"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020566    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc0f02d-3530-4047-92a3-3de9d636d97a" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020571    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc0f02d-3530-4047-92a3-3de9d636d97a" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020583    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b91c86-18ff-4eea-ba02-eefc1294426d" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020589    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b91c86-18ff-4eea-ba02-eefc1294426d" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2100ad5-18af-45ee-8be0-32a40de53e90" containerName="payload-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020601    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2100ad5-18af-45ee-8be0-32a40de53e90" containerName="payload-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020613    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020620    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b1b244-87d4-4164-9a1a-2dc3c8941408" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020625    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b1b244-87d4-4164-9a1a-2dc3c8941408" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020670    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b91c86-18ff-4eea-ba02-eefc1294426d" containerName="payload-pre-processing"
Sep 11 14:23:50.020666 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020677    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfe6d6b-49f4-428a-b22d-5f5d36a9363e" containerName="payload-processing"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020684    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c798c7-5445-4cb2-88e4-2b25478bd892" containerName="payload-pre-processing"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020690    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b1b244-87d4-4164-9a1a-2dc3c8941408" containerName="payload-pre-processing"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020695    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a0dfb4f-5789-44a6-8d5a-4cc0848a0376" containerName="payload-processing"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020701    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020707    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc0f02d-3530-4047-92a3-3de9d636d97a" containerName="payload-pre-processing"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020712    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2100ad5-18af-45ee-8be0-32a40de53e90" containerName="payload-processing"
Sep 11 14:23:50.021643 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.020717    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c53164-e1f3-4097-9440-033a5a2ad246" containerName="main"
Sep 11 14:23:50.025174 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.025154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:50.029501 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.029479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:23:50.029635 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.029479    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:23:50.034198 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.034163    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"]
Sep 11 14:23:50.090005 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.089970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e440fce-2c16-4bf8-9814-9f0c44b2e19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq\" (UID: \"4e440fce-2c16-4bf8-9814-9f0c44b2e19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:50.191149 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.191097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e440fce-2c16-4bf8-9814-9f0c44b2e19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq\" (UID: \"4e440fce-2c16-4bf8-9814-9f0c44b2e19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:50.191713 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.191690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e440fce-2c16-4bf8-9814-9f0c44b2e19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq\" (UID: \"4e440fce-2c16-4bf8-9814-9f0c44b2e19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:50.335231 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.335195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:50.615498 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:50.615413    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"]
Sep 11 14:23:50.619099 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:23:50.619068    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e440fce_2c16_4bf8_9814_9f0c44b2e19d.slice/crio-cecdbf8d4bf21d0416e66698fdbc7db77a68d63f8df8140cbe8324781e816744 WatchSource:0}: Error finding container cecdbf8d4bf21d0416e66698fdbc7db77a68d63f8df8140cbe8324781e816744: Status 404 returned error can't find the container with id cecdbf8d4bf21d0416e66698fdbc7db77a68d63f8df8140cbe8324781e816744
Sep 11 14:23:51.264955 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:51.264921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq" event={"ID":"4e440fce-2c16-4bf8-9814-9f0c44b2e19d","Type":"ContainerStarted","Data":"f21acae0452c45badcf7c40e9b25bb30e25cc6c64b07d221804b16c0c9176913"}
Sep 11 14:23:51.265350 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:51.264974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq" event={"ID":"4e440fce-2c16-4bf8-9814-9f0c44b2e19d","Type":"ContainerStarted","Data":"cecdbf8d4bf21d0416e66698fdbc7db77a68d63f8df8140cbe8324781e816744"}
Sep 11 14:23:51.282754 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:51.282705    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq" podStartSLOduration=1.282689073 podStartE2EDuration="1.282689073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:51.282610191 +0000 UTC m=+1879.650199638" watchObservedRunningTime="2026-09-11 14:23:51.282689073 +0000 UTC m=+1879.650278515"
Sep 11 14:23:52.271131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:52.271093    2561 generic.go:356] "Generic (PLEG): container finished" podID="4e440fce-2c16-4bf8-9814-9f0c44b2e19d" containerID="f21acae0452c45badcf7c40e9b25bb30e25cc6c64b07d221804b16c0c9176913" exitCode=0
Sep 11 14:23:52.271560 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:52.271138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq" event={"ID":"4e440fce-2c16-4bf8-9814-9f0c44b2e19d","Type":"ContainerDied","Data":"f21acae0452c45badcf7c40e9b25bb30e25cc6c64b07d221804b16c0c9176913"}
Sep 11 14:23:53.404576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:53.404549    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:53.514647 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:53.514601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e440fce-2c16-4bf8-9814-9f0c44b2e19d-maas-api-service-ca\") pod \"4e440fce-2c16-4bf8-9814-9f0c44b2e19d\" (UID: \"4e440fce-2c16-4bf8-9814-9f0c44b2e19d\") "
Sep 11 14:23:53.514997 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:53.514976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e440fce-2c16-4bf8-9814-9f0c44b2e19d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e440fce-2c16-4bf8-9814-9f0c44b2e19d" (UID: "4e440fce-2c16-4bf8-9814-9f0c44b2e19d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:23:53.615709 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:53.615665    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e440fce-2c16-4bf8-9814-9f0c44b2e19d-maas-api-service-ca\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:23:54.281071 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:54.280999    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq" event={"ID":"4e440fce-2c16-4bf8-9814-9f0c44b2e19d","Type":"ContainerDied","Data":"cecdbf8d4bf21d0416e66698fdbc7db77a68d63f8df8140cbe8324781e816744"}
Sep 11 14:23:54.281071 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:54.281025    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"
Sep 11 14:23:54.281071 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:23:54.281049    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cecdbf8d4bf21d0416e66698fdbc7db77a68d63f8df8140cbe8324781e816744"
Sep 11 14:24:00.711582 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.711547    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"]
Sep 11 14:24:00.712000 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.711796    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj" podUID="5c2af385-e8ff-4d9f-a86d-051290c30ccf" containerName="payload-processing" containerID="cri-o://310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72" gracePeriod=30
Sep 11 14:24:00.717573 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.717543    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"]
Sep 11 14:24:00.717766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.717744    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr" podUID="b847e943-f370-4e26-98f1-3f96bd84a884" containerName="payload-pre-processing" containerID="cri-o://68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574" gracePeriod=30
Sep 11 14:24:00.930089 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.930057    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"]
Sep 11 14:24:00.932936 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.932899    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-261f9c-5g7zq"]
Sep 11 14:24:00.980945 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.980924    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:24:00.983658 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:00.983638    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:24:01.077342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.077307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xht5f\" (UniqueName: \"kubernetes.io/projected/5c2af385-e8ff-4d9f-a86d-051290c30ccf-kube-api-access-xht5f\") pod \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") "
Sep 11 14:24:01.077342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.077346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2af385-e8ff-4d9f-a86d-051290c30ccf-plugins-config-volume\") pod \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\" (UID: \"5c2af385-e8ff-4d9f-a86d-051290c30ccf\") "
Sep 11 14:24:01.077559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.077370    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfgbn\" (UniqueName: \"kubernetes.io/projected/b847e943-f370-4e26-98f1-3f96bd84a884-kube-api-access-pfgbn\") pod \"b847e943-f370-4e26-98f1-3f96bd84a884\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") "
Sep 11 14:24:01.077559 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.077424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b847e943-f370-4e26-98f1-3f96bd84a884-plugins-config-volume\") pod \"b847e943-f370-4e26-98f1-3f96bd84a884\" (UID: \"b847e943-f370-4e26-98f1-3f96bd84a884\") "
Sep 11 14:24:01.077802 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.077771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2af385-e8ff-4d9f-a86d-051290c30ccf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c2af385-e8ff-4d9f-a86d-051290c30ccf" (UID: "5c2af385-e8ff-4d9f-a86d-051290c30ccf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:24:01.077905 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.077772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b847e943-f370-4e26-98f1-3f96bd84a884-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b847e943-f370-4e26-98f1-3f96bd84a884" (UID: "b847e943-f370-4e26-98f1-3f96bd84a884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:24:01.079496 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.079476    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2af385-e8ff-4d9f-a86d-051290c30ccf-kube-api-access-xht5f" (OuterVolumeSpecName: "kube-api-access-xht5f") pod "5c2af385-e8ff-4d9f-a86d-051290c30ccf" (UID: "5c2af385-e8ff-4d9f-a86d-051290c30ccf"). InnerVolumeSpecName "kube-api-access-xht5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:24:01.079580 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.079535    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b847e943-f370-4e26-98f1-3f96bd84a884-kube-api-access-pfgbn" (OuterVolumeSpecName: "kube-api-access-pfgbn") pod "b847e943-f370-4e26-98f1-3f96bd84a884" (UID: "b847e943-f370-4e26-98f1-3f96bd84a884"). InnerVolumeSpecName "kube-api-access-pfgbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:24:01.178359 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.178324    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b847e943-f370-4e26-98f1-3f96bd84a884-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:24:01.178359 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.178356    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xht5f\" (UniqueName: \"kubernetes.io/projected/5c2af385-e8ff-4d9f-a86d-051290c30ccf-kube-api-access-xht5f\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:24:01.178551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.178370    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2af385-e8ff-4d9f-a86d-051290c30ccf-plugins-config-volume\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:24:01.178551 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.178384    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgbn\" (UniqueName: \"kubernetes.io/projected/b847e943-f370-4e26-98f1-3f96bd84a884-kube-api-access-pfgbn\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:24:01.312738 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.312706    2561 generic.go:356] "Generic (PLEG): container finished" podID="5c2af385-e8ff-4d9f-a86d-051290c30ccf" containerID="310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72" exitCode=0
Sep 11 14:24:01.312953 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.312750    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"
Sep 11 14:24:01.312953 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.312796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj" event={"ID":"5c2af385-e8ff-4d9f-a86d-051290c30ccf","Type":"ContainerDied","Data":"310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72"}
Sep 11 14:24:01.312953 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.312854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj" event={"ID":"5c2af385-e8ff-4d9f-a86d-051290c30ccf","Type":"ContainerDied","Data":"f38bf494231b6afb6f8dcf12185a014a37a449581f8c99077d02061aae36b83e"}
Sep 11 14:24:01.312953 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.312884    2561 scope.go:117] "RemoveContainer" containerID="310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72"
Sep 11 14:24:01.314231 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.314209    2561 generic.go:356] "Generic (PLEG): container finished" podID="b847e943-f370-4e26-98f1-3f96bd84a884" containerID="68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574" exitCode=0
Sep 11 14:24:01.314296 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.314258    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"
Sep 11 14:24:01.314346 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.314292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr" event={"ID":"b847e943-f370-4e26-98f1-3f96bd84a884","Type":"ContainerDied","Data":"68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574"}
Sep 11 14:24:01.314385 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.314369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr" event={"ID":"b847e943-f370-4e26-98f1-3f96bd84a884","Type":"ContainerDied","Data":"8c51c6ce67e9b37960afac1401782b02372f382fbed504fd69887afac7a85090"}
Sep 11 14:24:01.323614 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.323593    2561 scope.go:117] "RemoveContainer" containerID="310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72"
Sep 11 14:24:01.324133 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:24:01.324109    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72\": container with ID starting with 310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72 not found: ID does not exist" containerID="310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72"
Sep 11 14:24:01.324214 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.324144    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72"} err="failed to get container status \"310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72\": rpc error: code = NotFound desc = could not find container \"310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72\": container with ID starting with 310c4fed978125bcc964bde4eaec1e5bd8e27ef216b5a0df3fe72b04fa690c72 not found: ID does not exist"
Sep 11 14:24:01.324214 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.324168    2561 scope.go:117] "RemoveContainer" containerID="68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574"
Sep 11 14:24:01.331985 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.331964    2561 scope.go:117] "RemoveContainer" containerID="68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574"
Sep 11 14:24:01.332261 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:24:01.332242    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574\": container with ID starting with 68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574 not found: ID does not exist" containerID="68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574"
Sep 11 14:24:01.332318 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.332267    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574"} err="failed to get container status \"68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574\": rpc error: code = NotFound desc = could not find container \"68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574\": container with ID starting with 68871eea2499b80eaf9308df775321a54da1f261d507b883286267e4eeb01574 not found: ID does not exist"
Sep 11 14:24:01.345764 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.345742    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"]
Sep 11 14:24:01.348033 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.348013    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-261f9c-7586f6f79c-kklxj"]
Sep 11 14:24:01.363444 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.363421    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"]
Sep 11 14:24:01.367272 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:01.367252    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-261f9c-8497b9c6d6-qfrxr"]
Sep 11 14:24:02.116676 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:02.116645    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e440fce-2c16-4bf8-9814-9f0c44b2e19d" path="/var/lib/kubelet/pods/4e440fce-2c16-4bf8-9814-9f0c44b2e19d/volumes"
Sep 11 14:24:02.117051 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:02.117006    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c2af385-e8ff-4d9f-a86d-051290c30ccf" path="/var/lib/kubelet/pods/5c2af385-e8ff-4d9f-a86d-051290c30ccf/volumes"
Sep 11 14:24:02.117315 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:02.117301    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b847e943-f370-4e26-98f1-3f96bd84a884" path="/var/lib/kubelet/pods/b847e943-f370-4e26-98f1-3f96bd84a884/volumes"
Sep 11 14:24:03.601241 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:03.601208    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:24:13.722359 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:13.722320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:24:32.468449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:24:32.468414    2561 scope.go:117] "RemoveContainer" containerID="c8a150123fa6e2701554a8a3c828805ae4f66cc38477adddb4ce75a43889799c"
Sep 11 14:25:33.714260 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:25:33.714217    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:25:58.090284 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:25:58.090247    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:26:05.624023 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:26:05.623984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:26:12.900735 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:26:12.900701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:26:21.293035 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:26:21.292987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:26:32.506599 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:26:32.506559    2561 scope.go:117] "RemoveContainer" containerID="89a30de86a33ee1e60ea1ccf44025cabd84822d8584b877cd9577950bfb42492"
Sep 11 14:26:45.002970 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:26:45.002934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:26:53.012822 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:26:53.012784    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:27:32.233435 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:27:32.233400    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:27:32.239522 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:27:32.239501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z96fx_da5ddc0b-7a20-4aae-9003-49fb9f60661b/ovn-acl-logging/0.log"
Sep 11 14:28:20.888700 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:20.888664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:28:28.991895 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:28.991809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:28:35.189082 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:35.189045    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:28:45.403098 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403062    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"]
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403333    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b847e943-f370-4e26-98f1-3f96bd84a884" containerName="payload-pre-processing"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403345    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b847e943-f370-4e26-98f1-3f96bd84a884" containerName="payload-pre-processing"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403356    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2af385-e8ff-4d9f-a86d-051290c30ccf" containerName="payload-processing"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403365    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2af385-e8ff-4d9f-a86d-051290c30ccf" containerName="payload-processing"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403380    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e440fce-2c16-4bf8-9814-9f0c44b2e19d" containerName="revoke-keys"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403386    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e440fce-2c16-4bf8-9814-9f0c44b2e19d" containerName="revoke-keys"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403430    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b847e943-f370-4e26-98f1-3f96bd84a884" containerName="payload-pre-processing"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403443    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e440fce-2c16-4bf8-9814-9f0c44b2e19d" containerName="revoke-keys"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403453    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd10f49-bd76-441e-abf6-5b0d8c112275" containerName="istio-proxy"
Sep 11 14:28:45.403576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.403459    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2af385-e8ff-4d9f-a86d-051290c30ccf" containerName="payload-processing"
Sep 11 14:28:45.406323 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.406306    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.409359 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.409332    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-77c6fd9a-kserve-self-signed-certs\""
Sep 11 14:28:45.420830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.420807    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"]
Sep 11 14:28:45.457239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.457053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-dshm\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.457239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.457098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tmp-dir\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.457239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.457182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-model-cache\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.457239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.457220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-home\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.457239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.457244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tls-certs\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.457239 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.457273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p555v\" (UniqueName: \"kubernetes.io/projected/d1630e9c-568b-4da5-939b-0c32af2f5d8f-kube-api-access-p555v\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-model-cache\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558217 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-home\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558480 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tls-certs\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558480 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p555v\" (UniqueName: \"kubernetes.io/projected/d1630e9c-568b-4da5-939b-0c32af2f5d8f-kube-api-access-p555v\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558480 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-dshm\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558480 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tmp-dir\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558699 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-model-cache\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558699 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-home\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.558783 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.558707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tmp-dir\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.560515 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.560496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-dshm\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.560952 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.560931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tls-certs\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.583692 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.583652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p555v\" (UniqueName: \"kubernetes.io/projected/d1630e9c-568b-4da5-939b-0c32af2f5d8f-kube-api-access-p555v\") pod \"e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") " pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.715728 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.715629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:45.992609 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:45.992522    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"]
Sep 11 14:28:45.996997 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:28:45.996969    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1630e9c_568b_4da5_939b_0c32af2f5d8f.slice/crio-b8bf934abd59f5eee7933965df80c9cc9f1d306e07955eff67504a1d68ff6464 WatchSource:0}: Error finding container b8bf934abd59f5eee7933965df80c9cc9f1d306e07955eff67504a1d68ff6464: Status 404 returned error can't find the container with id b8bf934abd59f5eee7933965df80c9cc9f1d306e07955eff67504a1d68ff6464
Sep 11 14:28:46.479261 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:46.479222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq" event={"ID":"d1630e9c-568b-4da5-939b-0c32af2f5d8f","Type":"ContainerStarted","Data":"fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610"}
Sep 11 14:28:46.479684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:46.479270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq" event={"ID":"d1630e9c-568b-4da5-939b-0c32af2f5d8f","Type":"ContainerStarted","Data":"b8bf934abd59f5eee7933965df80c9cc9f1d306e07955eff67504a1d68ff6464"}
Sep 11 14:28:46.504032 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:46.503975    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq" podStartSLOduration=1.50395486 podStartE2EDuration="1.50395486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:46.502101263 +0000 UTC m=+2174.869690705" watchObservedRunningTime="2026-09-11 14:28:46.50395486 +0000 UTC m=+2174.871544303"
Sep 11 14:28:47.988955 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:47.988898    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:28:55.716817 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.716777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:55.717337 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.716839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:55.729472 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.729442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:55.785518 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.785481    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"]
Sep 11 14:28:55.789254 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.789236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.792506 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.792475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-77c6fd9a-kserve-self-signed-certs\""
Sep 11 14:28:55.802299 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.802264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"]
Sep 11 14:28:55.829917 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.829874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-dshm\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.830094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.829934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcb28\" (UniqueName: \"kubernetes.io/projected/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-kube-api-access-wcb28\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.830094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.830022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tmp-dir\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.830094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.830050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-home\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.830094 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.830067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-model-cache\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.830414 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.830097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tls-certs\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.930519 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tmp-dir\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.930519 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-home\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.930772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-model-cache\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.930772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tls-certs\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.930772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-dshm\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.930772 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcb28\" (UniqueName: \"kubernetes.io/projected/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-kube-api-access-wcb28\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.931017 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.930993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-model-cache\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.931092 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.931057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tmp-dir\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.931092 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.931070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-home\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.932933 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.932897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-dshm\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.933220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.933202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tls-certs\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:55.939188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:55.939166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcb28\" (UniqueName: \"kubernetes.io/projected/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-kube-api-access-wcb28\") pod \"e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") " pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:56.099013 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:56.098971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:28:56.376205 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:56.376122    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"]
Sep 11 14:28:56.380253 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:28:56.380226    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7120bbe9_bc42_48a9_800e_ed7cbf8c1e27.slice/crio-fe3e27899ff83588081b371111e1ae64366a3e141b375f46d89f5126dbc9c0d2 WatchSource:0}: Error finding container fe3e27899ff83588081b371111e1ae64366a3e141b375f46d89f5126dbc9c0d2: Status 404 returned error can't find the container with id fe3e27899ff83588081b371111e1ae64366a3e141b375f46d89f5126dbc9c0d2
Sep 11 14:28:56.524125 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:56.524090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq" event={"ID":"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27","Type":"ContainerStarted","Data":"6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642"}
Sep 11 14:28:56.524330 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:56.524132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq" event={"ID":"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27","Type":"ContainerStarted","Data":"fe3e27899ff83588081b371111e1ae64366a3e141b375f46d89f5126dbc9c0d2"}
Sep 11 14:28:56.535703 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:56.535672    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:28:56.548537 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:56.548486    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq" podStartSLOduration=1.5484710179999999 podStartE2EDuration="1.548471018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:56.547172429 +0000 UTC m=+2184.914761871" watchObservedRunningTime="2026-09-11 14:28:56.548471018 +0000 UTC m=+2184.916060460"
Sep 11 14:28:58.694631 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:28:58.694583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:29:06.099815 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:06.099775    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:29:06.100232 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:06.099837    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:29:06.117938 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:06.117893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:29:06.579984 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:06.579955    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:29:36.421515 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:36.421473    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"]
Sep 11 14:29:37.394063 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:37.394022    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"]
Sep 11 14:29:39.518400 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:39.518351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:29:48.892336 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:48.892291    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:29:51.439861 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.439806    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq" podUID="d1630e9c-568b-4da5-939b-0c32af2f5d8f" containerName="main" containerID="cri-o://fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610" gracePeriod=45
Sep 11 14:29:51.680684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.680657    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:29:51.759569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.759479    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1630e9c-568b-4da5-939b-0c32af2f5d8f" containerID="fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610" exitCode=0
Sep 11 14:29:51.759569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.759524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq" event={"ID":"d1630e9c-568b-4da5-939b-0c32af2f5d8f","Type":"ContainerDied","Data":"fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610"}
Sep 11 14:29:51.759569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.759531    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"
Sep 11 14:29:51.759569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.759552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq" event={"ID":"d1630e9c-568b-4da5-939b-0c32af2f5d8f","Type":"ContainerDied","Data":"b8bf934abd59f5eee7933965df80c9cc9f1d306e07955eff67504a1d68ff6464"}
Sep 11 14:29:51.759569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.759569    2561 scope.go:117] "RemoveContainer" containerID="fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610"
Sep 11 14:29:51.767394 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.767375    2561 scope.go:117] "RemoveContainer" containerID="fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610"
Sep 11 14:29:51.767655 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:29:51.767637    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610\": container with ID starting with fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610 not found: ID does not exist" containerID="fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610"
Sep 11 14:29:51.767695 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.767667    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610"} err="failed to get container status \"fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610\": rpc error: code = NotFound desc = could not find container \"fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610\": container with ID starting with fcc5f521fb0c77e5b92943f8f79f284689c1a4b248c4d5e1db34e2a6911d0610 not found: ID does not exist"
Sep 11 14:29:51.798090 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798047    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-dshm\") pod \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") "
Sep 11 14:29:51.798206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tls-certs\") pod \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") "
Sep 11 14:29:51.798206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tmp-dir\") pod \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") "
Sep 11 14:29:51.798206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-model-cache\") pod \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") "
Sep 11 14:29:51.798206 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798190    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p555v\" (UniqueName: \"kubernetes.io/projected/d1630e9c-568b-4da5-939b-0c32af2f5d8f-kube-api-access-p555v\") pod \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") "
Sep 11 14:29:51.798380 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-home\") pod \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\" (UID: \"d1630e9c-568b-4da5-939b-0c32af2f5d8f\") "
Sep 11 14:29:51.798476 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1630e9c-568b-4da5-939b-0c32af2f5d8f" (UID: "d1630e9c-568b-4da5-939b-0c32af2f5d8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:51.798592 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798463    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1630e9c-568b-4da5-939b-0c32af2f5d8f" (UID: "d1630e9c-568b-4da5-939b-0c32af2f5d8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:51.798592 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.798535    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-home" (OuterVolumeSpecName: "home") pod "d1630e9c-568b-4da5-939b-0c32af2f5d8f" (UID: "d1630e9c-568b-4da5-939b-0c32af2f5d8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:51.800349 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.800326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-dshm" (OuterVolumeSpecName: "dshm") pod "d1630e9c-568b-4da5-939b-0c32af2f5d8f" (UID: "d1630e9c-568b-4da5-939b-0c32af2f5d8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:51.800449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.800384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1630e9c-568b-4da5-939b-0c32af2f5d8f" (UID: "d1630e9c-568b-4da5-939b-0c32af2f5d8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:29:51.800449 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.800393    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1630e9c-568b-4da5-939b-0c32af2f5d8f-kube-api-access-p555v" (OuterVolumeSpecName: "kube-api-access-p555v") pod "d1630e9c-568b-4da5-939b-0c32af2f5d8f" (UID: "d1630e9c-568b-4da5-939b-0c32af2f5d8f"). InnerVolumeSpecName "kube-api-access-p555v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:29:51.899615 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.899557    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:51.899615 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.899607    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:51.899615 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.899619    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:51.899615 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.899627    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:51.899905 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.899637    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p555v\" (UniqueName: \"kubernetes.io/projected/d1630e9c-568b-4da5-939b-0c32af2f5d8f-kube-api-access-p555v\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:51.899905 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:51.899646    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1630e9c-568b-4da5-939b-0c32af2f5d8f-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:52.083166 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.083119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"]
Sep 11 14:29:52.089845 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.089817    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-77c6fd9a-kserve-6c6d56c8bc-q7cxq"]
Sep 11 14:29:52.120840 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.120807    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1630e9c-568b-4da5-939b-0c32af2f5d8f" path="/var/lib/kubelet/pods/d1630e9c-568b-4da5-939b-0c32af2f5d8f/volumes"
Sep 11 14:29:52.412978 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.412879    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq" podUID="7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" containerName="main" containerID="cri-o://6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642" gracePeriod=45
Sep 11 14:29:52.652598 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.652574    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:29:52.766113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.766015    2561 generic.go:356] "Generic (PLEG): container finished" podID="7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" containerID="6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642" exitCode=0
Sep 11 14:29:52.766113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.766057    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"
Sep 11 14:29:52.766113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.766061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq" event={"ID":"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27","Type":"ContainerDied","Data":"6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642"}
Sep 11 14:29:52.766113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.766088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq" event={"ID":"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27","Type":"ContainerDied","Data":"fe3e27899ff83588081b371111e1ae64366a3e141b375f46d89f5126dbc9c0d2"}
Sep 11 14:29:52.766113 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.766116    2561 scope.go:117] "RemoveContainer" containerID="6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642"
Sep 11 14:29:52.773864 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.773844    2561 scope.go:117] "RemoveContainer" containerID="6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642"
Sep 11 14:29:52.774166 ip-10-0-128-105 kubenswrapper[2561]: E0911 14:29:52.774148    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642\": container with ID starting with 6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642 not found: ID does not exist" containerID="6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642"
Sep 11 14:29:52.774244 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.774174    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642"} err="failed to get container status \"6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642\": rpc error: code = NotFound desc = could not find container \"6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642\": container with ID starting with 6656f290d19245c830719d900f59e641239436e89565cc079b01de07cd391642 not found: ID does not exist"
Sep 11 14:29:52.805899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.805866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-model-cache\") pod \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") "
Sep 11 14:29:52.806023 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.805968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-dshm\") pod \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") "
Sep 11 14:29:52.806023 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.805996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tmp-dir\") pod \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") "
Sep 11 14:29:52.806023 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-home\") pod \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") "
Sep 11 14:29:52.806204 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tls-certs\") pod \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") "
Sep 11 14:29:52.806204 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcb28\" (UniqueName: \"kubernetes.io/projected/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-kube-api-access-wcb28\") pod \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\" (UID: \"7120bbe9-bc42-48a9-800e-ed7cbf8c1e27\") "
Sep 11 14:29:52.806204 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-model-cache" (OuterVolumeSpecName: "model-cache") pod "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" (UID: "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:52.806360 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806283    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" (UID: "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:52.806360 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-home" (OuterVolumeSpecName: "home") pod "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" (UID: "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:52.806360 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.806294    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-model-cache\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:52.808140 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.808110    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-dshm" (OuterVolumeSpecName: "dshm") pod "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" (UID: "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:29:52.808237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.808203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-kube-api-access-wcb28" (OuterVolumeSpecName: "kube-api-access-wcb28") pod "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" (UID: "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27"). InnerVolumeSpecName "kube-api-access-wcb28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:29:52.808237 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.808203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" (UID: "7120bbe9-bc42-48a9-800e-ed7cbf8c1e27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:29:52.907489 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.907448    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tls-certs\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:52.907489 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.907483    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcb28\" (UniqueName: \"kubernetes.io/projected/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-kube-api-access-wcb28\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:52.907489 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.907494    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-dshm\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:52.907731 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.907503    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-tmp-dir\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:52.907731 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:52.907513    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27-home\") on node \"ip-10-0-128-105.ec2.internal\" DevicePath \"\""
Sep 11 14:29:53.091442 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:53.091407    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"]
Sep 11 14:29:53.098387 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:53.098360    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-77c6fd9a-kserve-55c684bdd7-nllzq"]
Sep 11 14:29:54.115675 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:54.115640    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" path="/var/lib/kubelet/pods/7120bbe9-bc42-48a9-800e-ed7cbf8c1e27/volumes"
Sep 11 14:29:57.318117 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:29:57.318083    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:30:00.168059 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168019    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"]
Sep 11 14:30:00.168576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168412    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1630e9c-568b-4da5-939b-0c32af2f5d8f" containerName="main"
Sep 11 14:30:00.168576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168431    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1630e9c-568b-4da5-939b-0c32af2f5d8f" containerName="main"
Sep 11 14:30:00.168576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168447    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" containerName="main"
Sep 11 14:30:00.168576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168455    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" containerName="main"
Sep 11 14:30:00.168576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168526    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7120bbe9-bc42-48a9-800e-ed7cbf8c1e27" containerName="main"
Sep 11 14:30:00.168576 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.168539    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1630e9c-568b-4da5-939b-0c32af2f5d8f" containerName="main"
Sep 11 14:30:00.172781 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.172754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"
Sep 11 14:30:00.176848 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.176819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\""
Sep 11 14:30:00.182892 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.182870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"
Sep 11 14:30:00.190698 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.190673    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"]
Sep 11 14:30:00.471514 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.471423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"]
Sep 11 14:30:00.476384 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:30:00.476353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6259edbf_1b53_4c12_986c_dbd818f6da39.slice/crio-4590533f16eed86367df5c3481e680b8c3b42417ab811419988706fb0eb207b3 WatchSource:0}: Error finding container 4590533f16eed86367df5c3481e680b8c3b42417ab811419988706fb0eb207b3: Status 404 returned error can't find the container with id 4590533f16eed86367df5c3481e680b8c3b42417ab811419988706fb0eb207b3
Sep 11 14:30:00.801073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.800964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m" event={"ID":"6259edbf-1b53-4c12-986c-dbd818f6da39","Type":"ContainerStarted","Data":"64675b39442315cf82607064c1f86693cea586c3188ea9f303e703830ad3af10"}
Sep 11 14:30:00.801073 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.801021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m" event={"ID":"6259edbf-1b53-4c12-986c-dbd818f6da39","Type":"ContainerStarted","Data":"4590533f16eed86367df5c3481e680b8c3b42417ab811419988706fb0eb207b3"}
Sep 11 14:30:00.830613 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:00.830563    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m" podStartSLOduration=0.830549019 podStartE2EDuration="830.549019ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:00.830466853 +0000 UTC m=+2249.198056296" watchObservedRunningTime="2026-09-11 14:30:00.830549019 +0000 UTC m=+2249.198138461"
Sep 11 14:30:01.806778 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:01.806742    2561 generic.go:356] "Generic (PLEG): container finished" podID="6259edbf-1b53-4c12-986c-dbd818f6da39" containerID="64675b39442315cf82607064c1f86693cea586c3188ea9f303e703830ad3af10" exitCode=0
Sep 11 14:30:01.807202 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:01.806832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m" event={"ID":"6259edbf-1b53-4c12-986c-dbd818f6da39","Type":"ContainerDied","Data":"64675b39442315cf82607064c1f86693cea586c3188ea9f303e703830ad3af10"}
Sep 11 14:30:02.931940 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:02.931891    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"
Sep 11 14:30:03.815678 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:03.815650    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m"
Sep 11 14:30:03.815678 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:03.815661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-dj59m" event={"ID":"6259edbf-1b53-4c12-986c-dbd818f6da39","Type":"ContainerDied","Data":"4590533f16eed86367df5c3481e680b8c3b42417ab811419988706fb0eb207b3"}
Sep 11 14:30:03.815930 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:03.815706    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4590533f16eed86367df5c3481e680b8c3b42417ab811419988706fb0eb207b3"
Sep 11 14:30:05.100251 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.100217    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:30:05.786031 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.785996    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"]
Sep 11 14:30:05.786257 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.786245    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6259edbf-1b53-4c12-986c-dbd818f6da39" containerName="cleanup"
Sep 11 14:30:05.786300 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.786258    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6259edbf-1b53-4c12-986c-dbd818f6da39" containerName="cleanup"
Sep 11 14:30:05.786337 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.786322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6259edbf-1b53-4c12-986c-dbd818f6da39" containerName="cleanup"
Sep 11 14:30:05.789141 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.789126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.794438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.794413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cmhl6\""
Sep 11 14:30:05.794438 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.794435    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:30:05.794633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.794466    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:30:05.794633 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.794413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 14:30:05.795624 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.795606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:30:05.808720 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.808691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866471d2-4cf1-40ce-b98c-c0f08f95794c-metrics-certs\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.808863 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.808756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866471d2-4cf1-40ce-b98c-c0f08f95794c-openshift-service-ca\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.808863 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.808786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866471d2-4cf1-40ce-b98c-c0f08f95794c-maas-api-tls\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.808863 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.808801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2x7f\" (UniqueName: \"kubernetes.io/projected/866471d2-4cf1-40ce-b98c-c0f08f95794c-kube-api-access-l2x7f\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.818479 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.818449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"]
Sep 11 14:30:05.909208 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.909165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866471d2-4cf1-40ce-b98c-c0f08f95794c-metrics-certs\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.909410 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.909246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866471d2-4cf1-40ce-b98c-c0f08f95794c-openshift-service-ca\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.909410 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.909283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866471d2-4cf1-40ce-b98c-c0f08f95794c-maas-api-tls\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.909410 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.909303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2x7f\" (UniqueName: \"kubernetes.io/projected/866471d2-4cf1-40ce-b98c-c0f08f95794c-kube-api-access-l2x7f\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.909964 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.909901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866471d2-4cf1-40ce-b98c-c0f08f95794c-openshift-service-ca\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.911698 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.911672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866471d2-4cf1-40ce-b98c-c0f08f95794c-maas-api-tls\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.911815 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.911741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866471d2-4cf1-40ce-b98c-c0f08f95794c-metrics-certs\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:05.922781 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:05.922759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2x7f\" (UniqueName: \"kubernetes.io/projected/866471d2-4cf1-40ce-b98c-c0f08f95794c-kube-api-access-l2x7f\") pod \"maas-api-6c8f9cff69-tkfgl\" (UID: \"866471d2-4cf1-40ce-b98c-c0f08f95794c\") " pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:06.099088 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:06.099051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:06.385067 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:06.385020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"]
Sep 11 14:30:06.387068 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:30:06.387045    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866471d2_4cf1_40ce_b98c_c0f08f95794c.slice/crio-990f7614a0733ee2caab54492ca3076152c29b4f16579555a8922a3830928094 WatchSource:0}: Error finding container 990f7614a0733ee2caab54492ca3076152c29b4f16579555a8922a3830928094: Status 404 returned error can't find the container with id 990f7614a0733ee2caab54492ca3076152c29b4f16579555a8922a3830928094
Sep 11 14:30:06.388444 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:06.388425    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:30:06.833609 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:06.833571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl" event={"ID":"866471d2-4cf1-40ce-b98c-c0f08f95794c","Type":"ContainerStarted","Data":"990f7614a0733ee2caab54492ca3076152c29b4f16579555a8922a3830928094"}
Sep 11 14:30:08.843357 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:08.843269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl" event={"ID":"866471d2-4cf1-40ce-b98c-c0f08f95794c","Type":"ContainerStarted","Data":"1318bde6badf440c448a20ed6a723b21b58c37f2a9a1ad0219997a693a649b58"}
Sep 11 14:30:08.843766 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:08.843414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:08.875830 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:08.875769    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl" podStartSLOduration=1.6806181900000001 podStartE2EDuration="3.875749718s" totalImagesPullingTime="2.195131528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:08.874933951 +0000 UTC m=+2257.242523385" watchObservedRunningTime="2026-09-11 14:30:08.875749718 +0000 UTC m=+2257.243339160"
Sep 11 14:30:11.803815 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:11.803777    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:30:14.853538 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:14.853507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c8f9cff69-tkfgl"
Sep 11 14:30:17.572015 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:17.571975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:30:24.415956 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:24.415904    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-btf8f"]
Sep 11 14:30:27.594478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:27.594441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9b89ddf5-jbscj_05bbb342-e3de-4680-bc6d-b004e33489df/authorino/0.log"
Sep 11 14:30:32.541533 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:32.541499    2561 scope.go:117] "RemoveContainer" containerID="f21acae0452c45badcf7c40e9b25bb30e25cc6c64b07d221804b16c0c9176913"
Sep 11 14:30:33.685481 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:33.685451    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-c4gdv_92d26e17-3357-48e0-810e-357662a5615e/copy-manifests/0.log"
Sep 11 14:30:33.704923 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:33.704885    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-c4gdv_92d26e17-3357-48e0-810e-357662a5615e/manager/0.log"
Sep 11 14:30:34.437377 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:34.437344    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c8f9cff69-tkfgl_866471d2-4cf1-40ce-b98c-c0f08f95794c/maas-api/0.log"
Sep 11 14:30:34.545224 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:34.545195    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818935-qkrl4_b76b5bd0-2014-447c-b333-2dc17d291bf2/cleanup/0.log"
Sep 11 14:30:34.654160 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:34.654134    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818950-dj59m_6259edbf-1b53-4c12-986c-dbd818f6da39/cleanup/0.log"
Sep 11 14:30:35.525701 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:35.525671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9b89ddf5-jbscj_05bbb342-e3de-4680-bc6d-b004e33489df/authorino/0.log"
Sep 11 14:30:35.874605 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:35.874575    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-j6mbw_c48d1677-1c7f-412c-a532-f870ebb388a2/kuadrant-console-plugin/0.log"
Sep 11 14:30:36.112131 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:36.112102    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-btf8f_d7f8fa63-feac-4144-b598-f7bff621a5d7/limitador/0.log"
Sep 11 14:30:36.833779 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:36.833752    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-jpqm6_e88d754d-c6e7-4bc6-8f7b-9bde58c5e995/kube-auth-proxy/0.log"
Sep 11 14:30:38.484108 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:38.484074    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2zr8g_866f5d39-dc4e-4a9b-b0ce-559b54cf2c69/main/0.log"
Sep 11 14:30:44.267900 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.267859    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5cfjv/must-gather-n79mk"]
Sep 11 14:30:44.274985 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.274962    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.281188 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.281157    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5cfjv\"/\"openshift-service-ca.crt\""
Sep 11 14:30:44.281300 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.281207    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5cfjv\"/\"default-dockercfg-jjc6s\""
Sep 11 14:30:44.282197 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.282173    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5cfjv\"/\"kube-root-ca.crt\""
Sep 11 14:30:44.286927 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.286887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cfjv/must-gather-n79mk"]
Sep 11 14:30:44.431040 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.431002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9rf\" (UniqueName: \"kubernetes.io/projected/6d4b6834-2352-46fd-9200-5958bd786000-kube-api-access-2n9rf\") pod \"must-gather-n79mk\" (UID: \"6d4b6834-2352-46fd-9200-5958bd786000\") " pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.431234 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.431046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6d4b6834-2352-46fd-9200-5958bd786000-must-gather-output\") pod \"must-gather-n79mk\" (UID: \"6d4b6834-2352-46fd-9200-5958bd786000\") " pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.532526 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.532438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9rf\" (UniqueName: \"kubernetes.io/projected/6d4b6834-2352-46fd-9200-5958bd786000-kube-api-access-2n9rf\") pod \"must-gather-n79mk\" (UID: \"6d4b6834-2352-46fd-9200-5958bd786000\") " pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.532526 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.532484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6d4b6834-2352-46fd-9200-5958bd786000-must-gather-output\") pod \"must-gather-n79mk\" (UID: \"6d4b6834-2352-46fd-9200-5958bd786000\") " pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.532794 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.532777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6d4b6834-2352-46fd-9200-5958bd786000-must-gather-output\") pod \"must-gather-n79mk\" (UID: \"6d4b6834-2352-46fd-9200-5958bd786000\") " pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.555102 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.555069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9rf\" (UniqueName: \"kubernetes.io/projected/6d4b6834-2352-46fd-9200-5958bd786000-kube-api-access-2n9rf\") pod \"must-gather-n79mk\" (UID: \"6d4b6834-2352-46fd-9200-5958bd786000\") " pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.584084 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.584035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cfjv/must-gather-n79mk"
Sep 11 14:30:44.866005 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.865968    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cfjv/must-gather-n79mk"]
Sep 11 14:30:44.868903 ip-10-0-128-105 kubenswrapper[2561]: W0911 14:30:44.868869    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d4b6834_2352_46fd_9200_5958bd786000.slice/crio-6abfaf644750e1d3c645b5cd926eaa64171e7e2dbb653a453688754e36f1a5b9 WatchSource:0}: Error finding container 6abfaf644750e1d3c645b5cd926eaa64171e7e2dbb653a453688754e36f1a5b9: Status 404 returned error can't find the container with id 6abfaf644750e1d3c645b5cd926eaa64171e7e2dbb653a453688754e36f1a5b9
Sep 11 14:30:44.990072 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:44.990032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/must-gather-n79mk" event={"ID":"6d4b6834-2352-46fd-9200-5958bd786000","Type":"ContainerStarted","Data":"6abfaf644750e1d3c645b5cd926eaa64171e7e2dbb653a453688754e36f1a5b9"}
Sep 11 14:30:45.997829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:45.997736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/must-gather-n79mk" event={"ID":"6d4b6834-2352-46fd-9200-5958bd786000","Type":"ContainerStarted","Data":"d977c9acef55f76fcf475a4cbc1b78bfc9cc12130ee989fc79b71d4c0497677f"}
Sep 11 14:30:45.997829 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:45.997786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/must-gather-n79mk" event={"ID":"6d4b6834-2352-46fd-9200-5958bd786000","Type":"ContainerStarted","Data":"32eca4fd0307e9e67dcb9cf49547abcf8e0a2557f6d6a763c780b98178974d0d"}
Sep 11 14:30:46.018869 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:46.018807    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5cfjv/must-gather-n79mk" podStartSLOduration=1.261913128 podStartE2EDuration="2.018788203s" totalImagesPullingTime="756.875075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:46.017203167 +0000 UTC m=+2294.384792607" watchObservedRunningTime="2026-09-11 14:30:46.018788203 +0000 UTC m=+2294.386377646"
Sep 11 14:30:47.401509 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:47.401472    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fp4jl_513fe987-6474-4973-b14e-14020d75bdd8/global-pull-secret-syncer/0.log"
Sep 11 14:30:47.880251 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:47.880209    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8t86n_707077db-11b9-4e88-9a4b-23b441e474fd/konnectivity-agent/0.log"
Sep 11 14:30:47.986223 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:47.986189    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-105.ec2.internal_694fbd47ec3c35fbb6cd78edfdf87922/haproxy/0.log"
Sep 11 14:30:52.803904 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:52.803860    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9b89ddf5-jbscj_05bbb342-e3de-4680-bc6d-b004e33489df/authorino/0.log"
Sep 11 14:30:53.067934 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:53.067880    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-j6mbw_c48d1677-1c7f-412c-a532-f870ebb388a2/kuadrant-console-plugin/0.log"
Sep 11 14:30:53.151734 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:53.151688    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-btf8f_d7f8fa63-feac-4144-b598-f7bff621a5d7/limitador/0.log"
Sep 11 14:30:55.266514 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:55.266474    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k5cwr_1d063270-9043-4db1-b7fc-eadb4dc7be0d/node-exporter/0.log"
Sep 11 14:30:55.302267 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:55.302221    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k5cwr_1d063270-9043-4db1-b7fc-eadb4dc7be0d/kube-rbac-proxy/0.log"
Sep 11 14:30:55.340478 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:55.340447    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k5cwr_1d063270-9043-4db1-b7fc-eadb4dc7be0d/init-textfile/0.log"
Sep 11 14:30:57.162461 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.162417    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"]
Sep 11 14:30:57.172569 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.172532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.185624 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.185592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"]
Sep 11 14:30:57.269088 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.269038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppb5\" (UniqueName: \"kubernetes.io/projected/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-kube-api-access-wppb5\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.269088 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.269091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-sys\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.269530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.269226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-proc\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.269530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.269274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-lib-modules\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.269530 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.269376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-podres\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.369890 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.369850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-podres\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.370209 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppb5\" (UniqueName: \"kubernetes.io/projected/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-kube-api-access-wppb5\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.370340 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-sys\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.370450 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-sys\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.370638 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-podres\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.370758 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-proc\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.370899 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-lib-modules\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.371105 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.371047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-lib-modules\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.371105 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.370540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-proc\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.381813 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.381776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppb5\" (UniqueName: \"kubernetes.io/projected/bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0-kube-api-access-wppb5\") pod \"perf-node-gather-daemonset-sshc2\" (UID: \"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.486354 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.486111    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:57.782240 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:57.781263    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"]
Sep 11 14:30:58.073793 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:58.073749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2" event={"ID":"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0","Type":"ContainerStarted","Data":"4d49ead2debe44f3232fd3cb922ff458577c48883705d15a924b78fd3c41638a"}
Sep 11 14:30:58.074160 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:58.074132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2" event={"ID":"bb3d1b58-cb48-4e5a-915f-bc8cb9d717f0","Type":"ContainerStarted","Data":"37cb57a58b7cdedb0dc5c692dec1efa74d1c99d026638fe1197138293b985ab5"}
Sep 11 14:30:58.074342 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:58.074327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:30:58.103377 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:30:58.103316    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2" podStartSLOduration=1.103295269 podStartE2EDuration="1.103295269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:58.102619496 +0000 UTC m=+2306.470208940" watchObservedRunningTime="2026-09-11 14:30:58.103295269 +0000 UTC m=+2306.470884712"
Sep 11 14:31:01.191220 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:01.191189    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dsn9c_b9d9a6af-284c-4150-926d-f0cd36705cba/dns/0.log"
Sep 11 14:31:01.231413 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:01.231381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dsn9c_b9d9a6af-284c-4150-926d-f0cd36705cba/kube-rbac-proxy/0.log"
Sep 11 14:31:01.479367 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:01.479294    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7h9r7_d45bb637-dafd-45e8-95dd-3731ba76f452/dns-node-resolver/0.log"
Sep 11 14:31:02.333684 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:02.333652    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g4phg_c5373e77-d971-4400-b35d-cd4bbe543ae5/node-ca/0.log"
Sep 11 14:31:04.389461 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:04.389432    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-jpqm6_e88d754d-c6e7-4bc6-8f7b-9bde58c5e995/kube-auth-proxy/0.log"
Sep 11 14:31:05.095515 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:05.095484    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-sshc2"
Sep 11 14:31:05.676832 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:05.676800    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fz5x6_36bcb94d-4a3b-476a-b64a-8531b0e398f5/serve-healthcheck-canary/0.log"
Sep 11 14:31:06.608476 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:06.608441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kf8c2_6725612c-a151-4d67-9b06-5bde83044106/kube-rbac-proxy/0.log"
Sep 11 14:31:06.648591 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:06.648545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kf8c2_6725612c-a151-4d67-9b06-5bde83044106/exporter/0.log"
Sep 11 14:31:06.723769 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:06.723738    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kf8c2_6725612c-a151-4d67-9b06-5bde83044106/extractor/0.log"
Sep 11 14:31:08.681333 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:08.681240    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-jpqm6_e88d754d-c6e7-4bc6-8f7b-9bde58c5e995/kube-auth-proxy/0.log"
Sep 11 14:31:10.324711 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:10.324678    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-c4gdv_92d26e17-3357-48e0-810e-357662a5615e/manager/0.log"
Sep 11 14:31:10.433126 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:10.433097    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-c4gdv_92d26e17-3357-48e0-810e-357662a5615e/copy-manifests/0.log"
Sep 11 14:31:11.000710 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:11.000676    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-jpqm6_e88d754d-c6e7-4bc6-8f7b-9bde58c5e995/kube-auth-proxy/0.log"
Sep 11 14:31:12.474269 ip-10-0-128-105 kubenswrapper[2561]: I0911 14:31:12.474240    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-594d9cdff7-96j95_3f0745ad-efa0-42c3-8a49-70d6640b20f1/manager/0.log"